1.

Introduction

Optical coherence tomography (OCT) is the optical equivalent of ultrasound, using light instead of sound to produce cross-sectional images of tissue. Resolutions up to 1 to $2\mu \mathrm{m}$ can be achieved, which are 10 to 25 times higher than high-frequency ultrasound and approach that of microscopy, thus allowing tissue differentiation based on morphology. OCT images range to approximately 2 mm in depth: a limitation mainly due to light scattering, which causes a decrease of OCT signal magnitude with increasing depth. This decay of OCT signal is directly related to the optical properties of the tissue and is quantified by the attenuation coefficient ${\mu }_{\mathrm{OCT}}$. Several studies have shown that the quantitative analysis of the OCT-signal attenuation allows in vivo differentiation between dissimilar tissue types, for example, in retinal imaging^1,2 and in cardiovascular plaque differentiation.^3–5 In addition, in the kidney,^6,7 in the upper urinary tract,⁸ in the bladder,⁹ and in axillary lymph node imaging,^10,11 quantitative analysis of the OCT-signal attenuation has shown differences between normal tissue and cancerous tissue. In the gynecological clinic, quantitative OCT has been shown to be helpful in distinguishing normal vulvar tissue from vulvar intraepithelial neoplasia (VIN).¹² VIN is a premalignant lesion that can develop into vulvar squamous cell carcinoma, which is the fourth most common gynecological type of cancer.¹³ VIN is diagnosed by punch biopsy, which is painful and may harm cosmetic appearance. Since VIN often recurs, and biopsies are required to determine the abnormality of the skin, some patients are required to undergo many biopsies throughout their lifetime. Quantitative analysis of the OCT signals may be an alternative method to distinguish VIN from normal vulvar skin.¹² In the quantitative analysis, a mathematical model based on Beer’s law containing the attenuation coefficient and accounting for OCT system parameters, is fitted to the OCT signal in a user-selected region of interest (ROI). Clearly, correct region selection is important for accurate quantification of the OCT signal-derived ${\mu }_{\mathrm{OCT}}$. Because OCT and quantitative attenuation analysis are increasingly used as diagnostic tools in clinical research, information on the observer differences and learning curves associated with this analysis is needed.

The aim of this study is to investigate the learning curve and the interobserver variance of quantitative OCT-imaging analysis of suspicious lesions of the vulva. Therefore, the ${\mu }_{\mathrm{OCT}}$ of normal and suspicious vulvar lesions (including VIN lesions) was determined from OCT images acquired earlier at five time points by three students and by three OCT experts, who form a reference consensus. Previous analysis¹² revealed a statistically significant difference between ${\mu }_{\mathrm{OCT}}$ of healthy ($2.1\pm 1.4{\mathrm{mm}}^{-1}$) and VIN ($6.2\pm 2.1{\mathrm{mm}}^{-1}$). We use the area under the curve of the receiver operating characteristic (ROC-AUC) as the primary metric for the learning of individual observers. In our previous study, a ROC-AUC of 0.95 was found. The interobserver differences between the three students are analyzed by a linear mixed effects model. Bland–Altman plots are used to compare the results of the three students to the consensus.

2.

Materials and Methods

2.2.

Optical Coherence Tomography Data Analysis

The OCT data were fitted with a single exponential decay model in which the amplitude ($A$) and decay coefficient (${\mu }_{\mathrm{OCT}}+{\mu }_{\mathrm{cal}}$) were the free running parameters and the offset noise ($y_0$) was fixed at the noise level derived from a region of the image where no tissue was present.^14,15 The term ${\mu }_{\mathrm{cal}}$ accounts for attenuation due to the OCT system itself: the confocal point spread function and sensitivity roll-off in depth.^14,15 This factor can be calibrated from an attenuation measurement on a very weakly scattering sample, in our case a 1000-fold dilution of Intralipid 20%. Note that using this method, water absorption is accounted for by ${\mu }_{\mathrm{cal}}$. In our model, ${\mu }_{\mathrm{OCT}}$ is a measure for the scattering of the tissue under study and a parameter to discriminate between normal vulvar tissue and diseased vulvar tissue. Within a cross-sectional OCT image, the epidermal layer appears as a dark gray band (Fig. 1). The OCT image of the tissue was straightened before selecting the ROI. In normal vulvar tissue, the investigator selected the ROI in this epidermal homogenous layer. In diseased vulvar tissue (with a visual suspicious lesion present), the ROI in the epidermal layer within the lesion was selected by the investigator. A digital record is stored for each analysis including geometrical coordinates of the chosen ROI in the image as well as all fit parameters and their 95% confidence intervals. This way, the OCT analysis can be reviewed a posteriori.

Fig. 1

Selection of the region of interest (ROI) and determination of accompanied ${\mu }_{\mathrm{OCT}}$ value are shown in normal vulvar tissue (a) and in diseased vulvar tissue (b). The ROI is selected with green (vertical and horizontal) lines from the epidermal layer. The average intensity versus depth is plotted in the line graphs and is calculated from data between the vertical green lines. The region from which the signal is fitted is depicted by the horizontal green lines.

2.3.

Study Design

OCT images were evaluated at five time points (T1, T2, T3, T4, V) by three students and a consensus team consisting of three experts [Daniel M. de Bruin (DMdB), Ronni Wessels (RW), and Dirk J. Fabers]. Prior to each time point, all images were renamed using nondescript random names to prevent name recognition. At T1, the expert team first excluded images with poor quality (e.g., low signal-to-noise ratio, out-of-focus, and saturation by specular reflection). Of the remaining 175 images (81 normal regions and 94 suspicious regions), consensus ${\mu }_{\mathrm{OCT}}$ values were determined. Next, three medical students, with no prior OCT experience, were introduced to this technique and taught how to interpret OCT images and how to select the ROI and calculate the ${\mu }_{\mathrm{OCT}}$ as described above. To gain familiarity with the OCT images and the analysis procedure, their assessment at T1 was performed on the complete dataset of 175 OCT images. The ${\mu }_{\mathrm{OCT}}$ was subsequently determined at T2, T3, and T4 separated by 2 to 3 days by all three students and the expert consensus. These assessments were performed on a 60-image, randomly selected subset of the original dataset of the 175 images. The expert team reviewed the dataset to include only images that showed either clear normal or suspicious regions, after which 54 OCT images remained (22 OCT images from 13 normal regions and 32 OCT images from 16 suspicious lesions). After the fourth assessment, a validation (V) was performed on a newly randomly selected 60-image subset. After expert review, this set contained 53 OCT images (22 OCT images from 14 normal regions and 31 OCT images from 19 suspicious lesions). Figure 2 shows the study design. The students and experts were blinded for tissue-type (normal or suspicious) throughout the whole study.

Fig. 2

Study design.

3.

Results

For each student and the consensus, ROC-AUCs were calculated. The medians and interquartile ranges (IQRs) of the ROC-AUCs for the three students and the consensus are presented in Table 1, Fig. 3, and Fig. 4. In Fig. 4, the solid line represents the median values, the boxes the IQRs, and the whiskers the smallest and the largest nonoutliers. Each data point outside the range of the whiskers is shown individually. The consensus has a very stable discrimination with accuracy around 0.80. Student 1 attained a greater accuracy at the last session (session 4) and the validation round (0.88 and 0.90, respectively). Student 2 performed quite well already at session 1 with an ROC-AUC of 0.78 (0.71 to 0.85), while the consensus had an ROC-AUC of 0.83 (0.77 to 0.89). Student 2 stayed at this level, but then performed worse on the validation set. Student 3 performed better at session 2 compared with 1, but declined in performance at sessions 3 and 4. At the validation set, however, ROC-AUC was 0.75 (0.69 to 0.82), which was only slightly worse than the consensus [ROC-AUC 0.81 (0.76 to 0.86)]. When looking at the validation session, students 1 and 3 performed better than at session 1, indicating that ${\mu }_{\mathrm{OCT}}$ determination can, in principle, be learned rapidly after a short training session. The ability to differentiate between normal vulvar tissue and diseased vulvar tissue is also shown in ROC curves at each time point. Figure 5 presents these ROC analyses for the three students and the consensus.

Table 1

The median [interquartile range (IQR)] ROC-AUCs for the three students and the consensus group at each time point (T1–T4) and the validation set. Evaluation at T1 is performed on a 175-image dataset; and T2–T4 on a 54-image dataset, which was randomly renamed between sessions. The validation set consisted of 53 new randomly drawn images.

Fig. 3

Learning curves. Graphical respresentation of the area under the curve of the receiver operating characteristic (ROC-AUC) of the consensus and students 1 to 3 at different time points shown in Table 1.

Fig. 4

The median [interquartile range (IQR)] ROC-AUCs for the three students and the consensus group at each time point (T1–T4) and the validation set. Box = IQR; whisker = smallest and largest nonoutliers. Data-points are the points outside the whiskers.

Fig. 5

ROC curves at each time point (T1–T4) and the validation assessment for the three students (S1–S3) and the consensus group (C).

The mixed effects models show that the mean ${\mu }_{\mathrm{OCT}}$-value determined by the consensus group and three students pooled together remained stable over the five assessments for both the normal vulvar tissue samples ($\text{means}=3.3/2.8/4.1/3.8/3.3{\mathrm{mm}}^{-1}$) and for the diseased tissue samples ($5.5/5.4/6.0/6.1/5.9{\mathrm{mm}}^{-1}$). The ${\mu }_{\mathrm{OCT}}$-values of normal vulvar and diseased vulvar tissues were significantly different at all five time points ($p<0.001$). The median within-lesion variance of the three students (without the consensus group) in observed ${\mu }_{\mathrm{OCT}}$ over the $T=I-V$ assessments for normal vulvar tissue was $4.1/3.2/2.8/4.1/3.8{\mathrm{mm}}^{-1}$; for diseased tissue, there was a notable reduction in variance in time $3.7/3.1/1.2/1.1/1.8{\mathrm{mm}}^{-1}$. The within-lesion ICC for normal tissue samples for the five assessment time points was 0.25/0.21/0.19/0.14/0.25. For the diseased tissue samples, it was slightly higher, with the exception of the final validation assessment, 0.37/0.34/0.44/0.46/0.18.

The difference in ${\mu }_{\mathrm{OCT}}$ values between the consensus and the students is presented using Bland–Altman plots (Fig. 6) at the five different time points. All three students appeared better at estimating high values and worse at estimating low values of ${\mu }_{\mathrm{OCT}}$.

Fig. 6

Bland–Altman plots showing the differences between the student’s (S1–S3) calculated ${\mu }_{\mathrm{OCT}}$ values and those of the consensus group at each of the four time points (T1–T4). The vertical axis depicts the difference between the ${\mu }_{\mathrm{OCT}}$ values of the student and the ${\mu }_{\mathrm{OCT}}$ values of the consensus group in ${\mathrm{mm}}^{-1}$. On the horizontal axis, the ${\mu }_{\mathrm{OCT}}$ values of the consensus group (in ${\mathrm{mm}}^{-1}$) are given.

4.

Discussion and Conclusion

The potential of OCT to discriminate normal from diseased tissues using quantification of optical properties has been demonstrated in a number of recent studies.^6,7,9,12,16 Preferential clinical application of this technique is during diagnostic or therapeutic intervention, operated by the physician, providing real-time accurate assessment of tissue status. Because the currently used method requires operator-based choices (most importantly, the selection of a ROI), assessment of operator-induced variance in the tissue assessment is of great importance. The IDEAL framework¹⁷ describes the stages through which innovative interventional techniques normally pass upon clinical introduction: Idea (proof of concept), Development, Exploration (learning), Assessment, and Long-term study (surveillance). IDEAL characterizes all of these stages and recommends study design types for each. For applications other than ophthalmology and cardiology, OCT is currently in the Development/Exploration stage, which makes it opportune to conduct a learning study.

In this study, we define “learning” as the increased ability to discriminate healthy from diseased tissue based on the OCT attenuation coefficient. For this purpose, we used the area under the ROC-AUC as the primary metric of learning, where higher values indicate better discrimination. From the ROC curves, we conclude that forming a consensus between experienced observers results in repeatable differentiation of normal vulvar tissue from diseased vulvar tissue based on ${\mu }_{\mathrm{OCT}}$ (ROC-AUC of $\sim 0.8$, see Table 1). This indicates that using expert consensus as a benchmark is indeed feasible in this type of learning study. Student 1 performed almost at the same level as the consensus (in assessments 2 and 4, even better). On the contrary, student 2 showed a decline in ROC-AUC, indicating less accurate tissue classification. Student 3’s results were constant, yet consistently below the consensus results. Interestingly, the performance of student 3 declined during the training period, but was again closer to the consensus in the validation round.

Assessments at time points T2, T3, T4 were performed on a fixed, 54-image subset of the primary dataset, where the images were randomly renamed between each session. An image-based “memorizing effect” may have taken place, leading to an artificial stabilization of the ${\mu }_{\mathrm{OCT}}$ (because students remember which ROI they selected previously). We expect this effect to be more pronounced for “diseased” images where the epithelial layers are thicker compared with normal epithelium. Thus “memorized” selection of the ROI is easier for diseased tissues. On the other hand, repeated assessment of gray-scale images as in this study may lead to “analysis fatigue” resulting in less accurate ROI determination which would particularly influence the smaller ROIs associated with normal tissue. Evidence of either, which are essentially psychological effects, are not clearly found from Table 1, Fig. 3, or Fig. 4, but can be investigated in future studies by repeated evaluation of carefully designed layered phantoms.¹⁴

We constructed linear mixed effects models to estimate means and variances between normal and diseased tissues. The mean ${\mu }_{\mathrm{OCT}}$ values, averaged over all observers (consensus + students 1 to 3), were stable over all time points for both normal and diseased tissues, which again indicates the value of consensus measurements. To further quantify the learning process, we analyzed the variance of the ${\mu }_{\mathrm{OCT}}$ values of the students alone. Of specific interest is the development from the last training session (T4) to the validation round (V) where the dataset under analysis was changed. For normal tissue, the within-tissue variance slightly decreased (from 4.1 to $3.8{\mathrm{mm}}^{-1}$), indicating convergence of the assessments over all observers (students 1 to 3). Correspondingly, the within-tissue ICC increased (the proportion of variance attributable to tissue variation), implying stable variance due to observer influence. On the other hand, the within-tissue variance for diseased tissue increased (from 1.1 to $1.8{\mathrm{mm}}^{-1}$), again indicating convergence of the assessment over all observers while ICC decreased from 0.46 to 0.18. The latter finding implies increased variance due to observer influence which we consider to be consistent with the decrease in the performance of student 2.

Bland–Altman plots were constructed to show the difference between each student’s assessment and consensus, versus consensus. For all students, the difference was higher for smaller values of the attenuation coefficient. Normal tissue samples generally showed smaller values of ${\mu }_{\mathrm{OCT}}$ than diseased samples; moreover, the ROI selection of normal tissue is complicated by the generally thinner epithelial layer. In time, student 1 showed improvement in assessing lower (normal) attenuation coefficients, resulting in overall improvement in discrimination as expressed by increased ROC-AUC. The performance of student 2 to assess lower (normal) values of ${\mu }_{\mathrm{OCT}}$ decreased in time. Interestingly, the ability of student 3 to assess high values of the attenuation coefficient (diseased tissue) slightly increased during the training rounds, whereas the ability to assess low values (normal tissue) slightly decreased leading to stable, but low, ROC-AUC over time. This overestimation of low values of ${\mu }_{\mathrm{OCT}}$ compared with the consensus values might be attributed to the fact that thinner layers have fewer pixels to which the signal can be fitted, therefore, the determination of the attenuation coefficient of thinner layers is less accurate than for thicker layers. We demonstrate this using an increasing layered stack of $50\text{-}\mu \mathrm{m}$-thick silicone elastomer-based optical phantom building blocks developed following the recipe described in Ref. 14. Using an 800-nm OCT system with an axial resolution of $4.5\mu \mathrm{m}$, we find that thinner phantoms give larger variances in determined attenuation coefficients (see Fig. 7).

Fig. 7

OCT image of a phantom with various layer thicknesses. The graph shows fitted attenuation coefficients versus layer thickness. The vertical axis depicts the ${\mu }_{\mathrm{OCT}}$ values with the standard deviation of the different phantoms in the upper picture. On the horizontal axis, the thickness of the phantoms is given.

5.

Implications and Perspective

Our study shows that OCT attenuation analysis can be learned after a few training and feedback sessions. All students improved from T1 (instruction) to T2. After T2, their results diverged, but from the decrease of the ICC, we see that operator-induced variance remains a dominating factor. For large-scale clinical studies (IDEAL stages 3 to 5), automatic analysis of OCT data will be a practical necessity given the large amount of data to be analyzed. Automatic analysis may in part overcome the difficulties described here. Several strategies are possible, ranging from segmentation algorithms borrowed from ophthalmic application to delineation of (thin) layers to “per-pixel” assessment of the attenuation coefficient as recently proposed by Vermeer et al.¹⁸ Improvement of the analysis using these approaches remains to be evaluated by extensive benchmarking against manual assessment, preferably by expert consensus.

We conclude that the technical procedure for ${\mu }_{\mathrm{OCT}}$ determination for tissue classification does not require extensive training since all observers improved in performance after one training and feedback cycle. Paramount, however, to reduce observer-induced variance, is accurate identification of suspected lesions within the OCT images. Analysis of diseased, often thicker, layers proved to be more accurate for novices compared with analysis of healthy, often thinner, tissue layers. Analysis of the latter tissues may be improved by automated approaches based on image segmentation¹⁹ and per-pixel attenuation analysis.¹⁸ For smaller-scale clinical studies (IDEAL stages 1 and 2), a consensus evaluation of OCT attenuation data is recommendable. Automation inherently overcomes user-induced variance, yet will require thorough validation. For large-scale studies (IDEAL stages 3 to 5), automatic analysis, as described in Ref. 18, becomes a practical necessity.