To address the problems of error and omission detection in remote sensing image detection caused by the diverse scale changes of remote sensing object scales and the abundant proportion of small-scale objects, as well as the global and dense distribution of remote sensing objects, a remote sensing image detection improvement method based on YOLOv5-S is proposed. First, according to the characteristics of remote sensing objects, the data enhancement strategy is adopted to expand the dataset samples for the characteristics of remote sensing objects to improve the generalization ability of the model. Second, the contextual transformer module is introduced to the backbone feature extraction network and the feature fusion network to ensure the local feature extraction capability while improving the global information acquisition capability of the model, making full use of the input contextual information and guiding the dynamic attention matrix learning to improve the visual representation ability. Third, based on the original model, a shallow detection scale is added, and then a multiscale complex fusion structure is adopted. Meanwhile, the K-means++ algorithm replaces the original K-means algorithm and then clusters 12 anchor box sizes. Fourth, the efficient intersection over union loss is used to improve the accuracy of the remote sensing object recognition prediction. In the experiment on the on two optical remote sensing image datasets, a comparison with several object detection algorithms based on convolutional neural network is made, the results show that the mAP@0.5 tested on the remote sensing datasets is higher than the original YOLOv5-S. Compared with other models, the detection efficiency is higher, and the problems of small-scale object detection in remote sensing image have been significantly improved.