The existence of the beam vector effect requires rigorous electromagnetic computational methods for the optimization of subwavelength diffractive optical elements (DOEs). To design DOEs with high performance, we proposed an optimization strategy combining a global optimization algorithm with the finite difference time domain (FDTD) method. First, scalar initial solutions are obtained by the Gerchberg–Saxton algorithm. Then, regarding the diffraction efficiency, the uniformity error and figure of merit are calculated by FDTD as evaluation functions, and the optimization of the initial solutions is implemented by improved particle swarm optimization and continues with the simulated annealing algorithm. After several numerical simulations, the analysis results confirm the robustness of the algorithm and the reduction of the computational cost. The experimental data demonstrate that the correlation of diffraction energy orders between the simulated design and fabricated sample are above 85%. This method efficiently improves the DOEs design.