The traditional fractional order total variational model has better results in denoising and maintaining texture details in infrared images. However, it is difficult to determine the order of fractional order differentiation in image processing so that the model has the best denoising effect. To solve this problem, a fractional order total variational infrared image denoising model incorporating a flower pollination particle swarm optimization (PSO) algorithm is proposed in this paper. The model combines the search advantages of the flower pollination optimization algorithm and the PSO algorithm. The maximization multiobjective equation is designed as the fitness function of the optimization algorithm. The optimal order of the fractional order total variational model is found adaptively according to different features in different regions of the infrared image. The experimental results show that the improved model not only achieves the adaptivity of the adaption of the fractional order of total variational model order but also effectively removes the noise and retains the texture structure of infrared images to the maximum extent.