The study of flowing Brownian particles finds numerous biomedical applications, ranging from blood flow analysis to diffusion research. A mathematical model for the correlation function of laser radiation scattered by flowing Brownian particles measured with fiber-based optical coherence tomography (OCT), which accounts for the effects of sample arm optics, is presented. It is shown that the parameters of an OCT optical system of any complexity can be taken into account by using the ABCD ray tracing matrix approach. Specifically, the impact of any optical system can be characterized by the changes in the effective beam radius, which replaces the Gaussian beam radius in the existing mathematical models of scattered radiation. It is shown that the validity of the developed ABCD matrix formalism is governed by the condition that the source coherence length is much smaller than the Rayleigh range in the sample. The predictions of the developed model are compared with previously published theories and with experimental data and agree well with the latter.