An invisible track bed defect classification method based on distributed optical fiber sensing data acquisition and an attention Transformer model mechanism under the frequency domain is proposed. The vibration sensing data contain structural safety information of vehicles, rails, track beds, etc., covering the entire time period and entire track area of subway operation. To classify the invisible track bed defect rapidly and accurately, the original vibration signals are first reduced by down-sampling and envelope signal extraction. According to the regular characteristics of different types of signals, an fast Fourier transform (FFT) Attention Transformer (FFT-Attn-Transformer) sequence feature extraction architecture with a high recognition accuracy is proposed for model training. The results demonstrate that the accuracy, precision, recall rate, and F1-score are all above 98% using the proposed model, and the recognition accuracy of the defect test area is 99.47%, which has extremely high stability and accuracy, providing an innovative and feasible idea for the lack of effective monitoring scheme for invisible track bed defects.