1.1.

Challenges in Deep Tissue Imaging

Biological imaging studies at the cellular level have long been performed almost exclusively on samples in Petri dishes¹ but are increasingly becoming capable of studying cells in their natural environment, i.e., in biological tissue.^2–4 Two main factors are enabling this transition: the invention of multiphoton microscopes, notably the two-photon microscope,^5,6 and adaptive optics.^7,8 While one-photon methods, especially confocal microscopy, have been and will continue to be immensely useful across a wide range of conditions, two-photon excited fluorescence (TPEF) microscopy is often the method of choice for imaging in tissue because it benefits from intrinsic optical sectioning, cellular resolution, high sensitivity, and high imaging rate. Importantly, it is resilient to image degradation from the light scattering in tissue thanks to the nonlinear dependence upon illumination intensity as well as the detection scheme, which singles out ballistic photons, which carry information about the sample and discriminates against scattered photons, which carry no spatial information. However, TPEF microscopy is not a universal enabler of deep tissue imaging; as ballistic photons are depleted exponentially, imaging depth is in practice limited to 1 mm.³ TPEF microscopy, therefore, faces the significant challenge of extending the imaging depth. Recent years have seen the attempts to extend imaging depth by adaptive optics-assisted TPEF microscopy,^9–14 spurred on by advances in wave front shapers, i.e., two-dimensional spatial light modulators (2-D-SLMs)¹⁵ and deformable mirrors (DMs).¹⁶ At the present state, it seems that aberrations incurred by ballistic photons can be well corrected by adaptive optics.^17,18 However, it is much less clear whether adaptive optics is capable of recovering more than just a small fraction of scattered photons.^19–22 So, adaptive optics may well also be limited by the depletion of ballistic photons. In view of these considerations, it is apparent that it is a significant challenge to acquire TPEF images from tissue depths where only scattered photons penetrate ($>1\mathrm{mm}$). The above challenges constitute the motivation for our work on the so-called “lensless endoscopes,” a concept that marries TPEF microscopy and adaptive optics in a new way and at the same time adds a new component, an optical waveguide. From our viewpoint, a lensless endoscope consists of a long waveguide, without any additional elements attached, capable of acquiring a multiphoton image of an object located at its tip, all the while retaining all the flexibility in excitation parameters of a free-space multiphoton microscope.

1.2.

Concept of Lensless Endoscopes

With the realization that the Maxwell equations are time-reversible,²³ the concept becomes quite simple. The wave front emitted by a coherent point source and coupled into a waveguide (the input wave front) emerges from the other end as a wave front (the output wave front) that bears no resemblance to the input wave front [Fig. 1(a)]. Nevertheless, if one lets a time-reversed copy of the output wave front propagate back through the same waveguide, it will resemble a time-reversed copy of the input wave front, in other words, it will focus on the original point source [Fig. 1(b)]. By extension, this means that one can associate each distal point with a proximal wave front, which, after propagation through the waveguide, will focus on that distal point [Fig. 1(c)]. This can be formalized with the transmission matrix formulation.^24–26 This means that spatial information—perhaps counter-intuitively—is not lost during propagation through the waveguide and that the spatial information (in other words, the image of the object) can be recovered, e.g., by point-scanning imaging, which is sketched in Fig. 1(d), but other wave front shaping methods, computational methods, or a combination of the two can also be used.^27,28 The above rationale holds for waveguides with many degrees of freedom, up to the spatial and Fourier filtering effects associated with coupling into the waveguide. We might also add that the above concept really is a generalization of the $4f$ imaging setup; in the $4f$ setup, the first lens performs a physical Fourier transform of the object, and the second lens performs the inverse Fourier transform, which reconstitutes the image of the object; in the generalization, the waveguide performs some complex transformation and the wave front shaping/computation stage performs the inverse of the same transformation, again, reconstituting the image. With this understanding of the concept, the visions of the so-called “lensless endoscopes” appear quite readily: (i) as a waveguide can transmit an image without any additional elements attached to the distal tip, there is nothing that increases the bulkiness of this fiber probe beyond the diameter of the waveguide itself; with a standard optical fiber of $125\mu \mathrm{m}$ outer diameter, this would allow for ultrathin imaging probes, potentially capable of noninvasive deep-tissue endoscopic imaging; (ii) standard optical fibers are highly flexible, and the lensless endoscope concept in principle works no matter the geometrical configuration of the waveguide, this flexibility could evidently be an advantage in many applications; (iii) in the lensless endoscope concept, one has complete control over the output wave front (by intermediary of the input wave front), which might enable enhancement of endoscopic deep-tissue imaging with the methods currently being developed for adaptive optics-assisted microscopy;^10,29 (iv) another consequence of the control over the output wave front might be photostimulation with shaped light in an endoscopic setting; and (v) finally, the waveguides with many degrees of freedom used in lensless endoscopes generally have very low nonlinearity; hence it can be expected that they will support high-intensity ultrashort pulses without distorting them through nonlinear effects. This could enable many kinds of nonlinear imaging modalities in endoscopes, which rely on ultrashort excitation pulses.

Fig. 1

Concept of lensless endoscopes. (a) The wave front emitted by a coherent point source is collected by and transmitted through a waveguide; the transmitted wave front bears no resemblance with the initial wave front. (b) A time-reversed copy of the transmitted wave front from (a) is sent back through the same waveguide; the transmitted wave front now comes to a focus at the position of the point source from (a). (c) By extension, one can associate any distal focus position ($X,Y,Z$) with a proximal wave front ${\mathrm{\Phi }}^{\text{proximal}}$. (d) Lensless endoscopy by point-scanning, using a wave front shaper to generate the sequence of ${\mathrm{\Phi }}_j^{\text{proximal}}$ that results in a sequence of $N_{\mathrm{pix}}$ distal focus positions ($X_j,Y_j,Z_j$). Detectors for fluorescent signal detection are not shown.

1.3.

Scope of the Paper

In this paper, we focus on lensless endoscopes that aim to achieve multiphoton, particularly two-photon, image contrast. The path that leads to the end goal of a working multiphoton lensless endoscope is not yet universally agreed upon and many challenges lie ahead. We consider multicore fibers (MCFs) as a necessary technology for its realization. However, this is not a foregone conclusion; multiphoton image contrast with multimode fibers (MMFs), particularly graded-index MMFs, shows promise as well (See Refs. 30 and 31 for the initial reports on focusing of ultrashort pulses and two-photon imaging through MMF; and see Ref. 32 for a review of MMF in the context of endoscopes).

In this paper, we will thus review the progress that has been made in lensless endoscopes based on MCF. This paper is organized as follows: in Sec. 2, we emphasize the merits of MCF as the waveguide in a lensless endoscope; in Sec. 3, we briefly review the most common ways of interfacing the MCF with free space through a wave front shaping device; in Sec. 4, we take stock of the results delivered so far by MCF-based lensless endoscopes; in Sec. 5, we discuss the outstanding challenges that still stand in the way of achieving the end goal, and wherever possible we point to the progress being made; finally, Sec. 6 summarizes and concludes.

2.1.

Low Coupling Between Cores

The MCFs that we have fabricated for our studies have all been optimized to exhibit minimal coupling between cores to assure that light injected into one core remains confined to only that core during propagation through the MCF. The coupling, or cross talk, between cores is a function of several MCF parameters: the mode profile, or the mode-field diameter (MFD); the propagation constants; and the distance between cores, or pitch $\mathrm{\Lambda }$. Examples of cross talk values are $<-30\mathrm{dB}/\mathrm{m}$ in an MCF with $\mathrm{\Lambda }=15\mu \mathrm{m}$ and $\mathrm{MFD}=3.0\mu \mathrm{m}$ [Fig. 2(a)];⁴⁴ and $<-20\mathrm{dB}/\mathrm{m}$ in a double-clad MCF with $\mathrm{\Lambda }=11.8\mu \mathrm{m}$ and $\mathrm{MFD}=3.6\mu \mathrm{m}$ [Fig. 2(b)].⁴⁵ The low-cross talk has several important convenient consequences as will become clear in the following.

Fig. 2

Examples of MCFs developed for lensless endoscopy. (a) MCF with low-cross talk and single-mode cores;⁴⁴ (b) MCF with low-cross talk, single-mode cores, and a multimode inner cladding;^45,46 (c) MCF with low-cross talk, aperiodically arranged single-mode cores;⁴⁷ (d) MCF with low-cross talk, aperiodically arranged single-mode cores, and a multimode inner cladding.

2.2.

Memory Effect

Due to the low coupling between the cores of the MCF, there is a strong correlation between the phase ${\phi }_i^{\text{proximal}}$ of the beamlet injected into core $i$ and the phase of the beamlet emitted from the same core at the other end ${\phi }_i^{\text{distal}}$. Additionally, the beamlets propagating in all other cores $j\ne i$ have no influence on ${\phi }_i^{\text{distal}}$. Indeed, ${\phi }_i^{\text{distal}}={\phi }_i^{\text{proximal}}+\delta {\phi }_i$ where ${\phi }_i^{\text{proximal}}$ is the phase of beamlet $i$ after its incidence on the wave front shaper; $\delta {\phi }_i$ are the static phase offsets stemming either from intrinsic differences in the effective refractive index between cores or from external perturbations like twists and bends to the MCF. With a phase calibration step, mentioned in Sec. 3.4, one can effectively measure the $\delta {\phi }_i$, and if one subsequently uses the wave front shaper to set the ${\phi }_i^{\mathrm{proximal}}=-\delta {\phi }_i$, one effectively obtains a flat wave front emitted from the MCF, ${\phi }_i^{\mathrm{distal}}={\phi }_i^{\mathrm{proximal}}+{\delta }_i=-\delta {\phi }_i+\delta {\phi }_i=0$. From this point, it becomes very intuitive to exert control over the emitted wave front ${\phi }_i^{\mathrm{distal}}(x_i,y_i),i=\mathrm{1,2},\dots ,N$. If one lets ${\phi }_i^{\text{proximal}}=-\delta {\phi }_i+{\phi }_i^{\text{desired}}$ then, logically, ${\phi }_i^{\text{distal}}={\phi }_i^{\text{desired}}$.

This behavior is often referred to as the “memory effect.” Originally described by Freund et al.,⁴⁸ the memory effect designated the phenomenon that the wave front transmitted through a multiply scattering medium remains correlated with the wave front transmitted for a different incident angle. The word is often used in the context of MCF, even though it is not strictly necessary to invoke the effect as described in the original paper to understand the functioning of MCFs. Instead it suffices to think of the MCF as a phase plate as we have done above. Only in the case where there is a significant coupling between the cores does it become necessary to depart from the simple phase-plate approach described above; with significant coupling, input–output correlations decrease, minimizing the memory effect.

Furthermore, with knowledge of the MCF core pattern, i.e., the core positions ($x_i,y_i$), it is straightforward to calculate the ${\phi }_i^{\text{desired}}$ necessary for obtaining simple intensity profiles; in particular, we take the example of the generation of a focus at ($X,Y,Z$)

2.3.

Dispersion

In optical fibers, dispersion is normally divided into two classes: modal dispersion and chromatic dispersion.

Modal dispersion is the phenomenon that different transverse modes of light in a waveguide have different propagation constants and travel at different velocities. This leads to a “group delay spread” between the different modes propagating in an MMF. In an MCF, there is no modal dispersion per se, since all cores are single mode. Even so, a similar phenomenon occurs in MCF because slight differences between the cores lead to a group delay spread of the fundamental modes propagating in different cores.

It is instructive to compare the group delay spreads in MMF with those in MCF. For step-index and graded-index type MMFs, the theoretical group velocities are⁴⁹

Importantly, it is technically possible to compensate the group delay spread in MCF because light in core $i$ stays in that core due to the low-cross talk. So even though the MCF reported in Ref. 50 permitted coherent combination of 100 fs pulses after 2 m of MCF, the MCFs in Figs. 2(a) and 2(b) can in principle be brought to the same level of performance with a suitable group delay compensation scheme which we will discuss in Sec. 4.4. Compensation of modal dispersion in MMF is of course theoretically possible, but technically this would be a very daunting task, which would require the prerequisite of being able to couple initially nonoverlapping beams into the overlapping eigenmodes of the MMF. This is what is known as modal multiplexing in the context of telecommunications, and low-loss modal multiplexing has so far only been reported for up to six modes at a time.^51,52

As for chromatic dispersion, it is the phenomenon that the different colors travel at different velocities. It results in a temporal broadening of a pulse propagating on a given transverse mode and depends on the second derivative with respect to frequency of the propagation constant $\beta$ of that mode. In MCFs with single-mode cores, light in all the cores travels on the fundamental transverse mode and, therefore, experiences the same chromatic dispersion ${\beta }_2$, up to a core-to-core variation $\mathrm{\Delta }{\beta }_2$ on the order ${10}^{-4}$ as was highlighted just above. In this case, it is sufficient to do a global precompensation of ${\beta }_2$, common to all cores, which can be done by standard methods (prism pairs, chirped mirrors) on the laser beam before it is incident on the wave front shaper. The uncompensated $\mathrm{\Delta }{\beta }_2$ only affects the pulse width $\tau$ to the second order, as can be seen from Ref. 53

2.4.

Double Cladding

In true endoscopic imaging, the waveguide serves two purposes, first, it transports excitation light to the sample; second, it collects fluorescent signal from the sample and transports it in the reverse direction. This would seem to put MCF at a disadvantage—due to the low core fill factor (the total area of the cores divided by the total area of the MCF), which is around 3% in the low-cross talk designs in Figs. 2(a) and 2(c), collection of backscattered fluorescence signal through the cores alone is extremely inefficient. Fortunately, this apparent disadvantage can be effectively negated by adding a double cladding to the MCF, cf., Figs. 2(b) and 2(d). In those two designs, the single-mode cores are contained in an “inner cladding” of silica, which is surrounded by a ring of air holes, an “outer cladding.” Due to the big refractive index difference between silica in the inner cladding and air in the outer cladding, the whole inner cladding is a multimode waveguide with very high NA and therefore very efficient at collecting backscattered fluorescence. The presence of the double cladding does not alter the single-mode cores embedded in the inner cladding. In the double-cladding design, the excitation and the collection of fluorescence become decoupled. The transport of excitation light through the single-mode cores takes place as it would in a single-cladding MCF; importantly, the entire surface of the inner cladding collects fluorescent signal, raising the collection efficiency to near unity.

2.5.

Birefringent MCFs

The MCFs displayed in Fig. 2 have all been designed with nonbirefringent cores, i.e., cores that have no preferred polarization orientation. Due to slight intrinsic imperfections, the cores act as randomly oriented waveplates; if one injects polarized light into all cores, then the cores will emit random polarization states.⁵⁴ With such MCFs in a lensless endoscope—whose concept relies on the interference between emitted beamlets—one would then require a wave front shaping scheme that controls horizontal and vertical polarization states, in order to make both states interfere constructively on the same ($X,Y,Z$). Dual-polarization wave front shaping is possible,^33,55,56 yet it is significantly more complex than the single-polarization wave front shaping, which remains the norm. However, a single-polarization wave front shaping scheme together with a nonbirefringent MCF makes the compromise that half of the light—the light in the “wrong” polarization state—is effectively wasted. A birefringent MCF with fast and slow axes of the cores aligned with one another would effectively solve the problem, allowing to work in the simpler, single-polarization wave front shaping scheme while avoiding wasting light to the “wrong” polarization state, since the MCF would maintain the polarization state of the injected light. Such an MCF has been reported by Stone et al.⁵⁷ The 98-core MCF necessitated the presence of stress rods to induce birefringence, which inevitably leads to space constraints, but the authors were able to keep the core pitch relatively low, $16\text{-}\mu \mathrm{m}$ along one direction, and $14\mu \mathrm{m}$ along the other direction—this is similar to the nonbirefringent MCF in Fig. 2(a). The cores had polarization extinction ratios of around 20 dB and the polarization orientation of each core did not vary by $>\pm 3\mathrm{deg}$. A lensless endoscope based on this birefringent MCF was reported by Kim et al.,⁵⁸ confirming the expected intensity increase in multiphoton imaging compared to nonbirefringent fibers.

3.1.

Single SLM

Using only a single SLM as the wave front shaping device in a lensless endoscope remains the most utilized approach.^{45,47,58–61} The SLM plane can be placed in a plane that is conjugated with the MCF proximal end face, which was done in the early reports.^44,59 However, it is advantageous to offset the SLM from the conjugate plane, as in Fig. 3(a), and to inscribe a microlens array on the SLM.⁴⁵ If instead the focal plane of this microlens array is conjugated with the MCF proximal end face, injection efficiency into the single-mode cores of the MCF can be significantly improved. This is due to the fact that the majority of the light incident on the SLM can then be directed into the cores. The phase mask on the SLM thus comprises as many segments (microlenses) as there are cores of the MCF, and these segments should be located on the grid defined by the core positions, scaled by the magnification factor of the telescope. For the periodic variety of MCF, Figs. 2(a) and 2(c), we have not found any significant departure from the periodic grid, which can consequently be defined by only two parameters, the pitch $\mathrm{\Lambda }$ and an overall angle, which eases initial alignment of the SLM. With the aperiodic-type MCFs, Figs. 2(c) and 2(d), no such simplification is possible and all core positions must be known.

Fig. 3

Different wave front shaping approaches. (a) Wave front shaping with a single SLM in an MCF-based lensless endoscope. (b) Wave front shaping with a beam-scanning device and an SLM in an MCF-based lensless endoscope. (c) Wave front shaping with two SLMs in an MCF-based lensless endoscope.

In this approach, it is the SLM alone that controls the wave front ${\phi }_i^{\text{proximal}}(x_i,y_i)$ that gets injected into the MCF. This approach as principally limited by the update rate of the SLM. The SLM typically has to be a liquid-crystal-based device with a high number of pixels in order to have sufficient complexity to inscribe the microlens array, and the update rate of these devices is inherently limited by the reorientation time of the liquid crystals to typically around 10 Hz, although some variants exist that allow update rates around 100 Hz.

3.2.

Single SLM and Beam-Scanning Device

The wave front shaping approach based on a single SLM, described above, can often be enhanced by the addition of a beam-scanning device, such as a pair of galvanometric scan mirrors.⁴⁴ As shown in Fig. 3(b), the scan mirrors can be inserted in a plane that is conjugated with the MCF proximal end face. The action of the scan mirrors is then to add a tip and tilt to the input wave front, i.e., the terms ${\phi }_x^{(1)}$ and ${\phi }_y^{(1)}$ in Eq. (1), and the SLM is thus liberated from performing this task. This permits to take advantage of the memory effect in MCFs (cf., Sec. 2.2) to realize video-rate imaging (cf., Sec. 4.2).

The beam-scanning device does not have to be a pair of resonant galvanometric scan mirrors; nonresonant scan mirrors and, also, acousto-optic beam deflectors can be used which, incidentally, would provide for random-access scanning.

3.3.

Two SLMs

In addition to the approaches described above, the addition of a second SLM in the wave front shaping approach can give access to additional parameters of the wave front.⁴⁶ As sketched in Fig. 3(c), this is facilitated by having access to the Fourier plane between SLM1 and SLM2; SLM1 can “group” beamlets together in subgroups that come to a focus on the same point in the Fourier plane where one can manipulate subgroups independently of one another. This approach can also optionally be combined with a beam scanning device although not shown in Fig. 3(c). We typically use a DM in the place of SLM1 and a liquid crystal-based SLM in the place of SLM2; this allows the setup to take advantage of the fast response time of DMs, typically several kHz.

This wave front shaping approach permits to achieve control over the group delay of the input wave front, cf., Sec. 4.4.

3.4.

Phase Calibration Step

Before operation of the MCF-based lensless endoscope can commence, the unknown phases $\delta {\phi }_i$ must be measured. This is what we refer to as the phase calibration step. Numerous methods are available. A simple method, whose principle is, however, representative for the totality of methods, used in Ref. 44 first brings the emissions of all $N$ cores are in constructive interference at a point with known coordinates (0, 0, $Z$). The center core (core 0) and core $i$ are illuminated; the intensity at (0, 0, $Z$) is measured as a function of ${\phi }_i^{\text{proximal}}$; and ${\phi }_i^{\text{proximal}}$ is set at the value that gives maximum intensity; this is done for all pairs (0, $i$). At the end of the process, once all cores are illuminated with the calibrated ${\phi }_i^{\text{proximal}}$, a bright focus appears at (0, 0, $Z$). At this point, one also knows the distal phase since

In Ref. 45, a faster version of this method was used, where a spatial Fourier transform replaces the need to scan over ${\phi }_i^{\text{proximal}}$. Also, an even faster method is based on digital phase conjugation,^61,62 which additionally eliminates the need for sequential, pairwise illumination of cores, but requires an external reference beam.

4.1.

Imaging

Thompson et al.⁵⁹ were the first to report imaging with a lensless endoscope based on a waveguide with multiple cores. The authors used a single-SLM approach, cf., Sec. 3.1 to control the phase of light propagating in a 500-core subset of a waveguide containing a total of 30,000 cores. The authors were able to acquire transmission images of a United States Air Force (USAF) test chart over an $80\text{-}\mu \mathrm{m}$ field-of-view and could also demonstrate 3-D control of the focal spot. The waveguide that was employed was of a class typically called “imaging fiber bundle,” which is a class of MCF that is available commercially. However, commercial imaging fiber bundles are typically fabricated to meet the demands of applications in incoherent imaging, unlike the MCFs in Fig. 2, which have been optimized for lensless endoscopes. Imaging fiber bundles are typically used in applications where they transport an intensity image⁶³ or in confocal endomicroscopy.⁶⁴ More recently, speckle-based imaging has also been reported.^65,66 Imaging fiber bundles are often fabricated with very small $\mathrm{\Lambda }$, down to $5\mu \mathrm{m}$, which comes at the cost of increased cross talk between the cores and eliminates many of the advantages of MCF, cf., Sec. 2, but can often be tolerated in incoherent imaging.

4.2.

Video-Rate Imaging

In 2013, we realized a lensless endoscope employing a custom-designed MCF that we had optimized for application in lensless endoscopy,⁴⁴ the MCF seen in Fig. 2(a). This MCF, with its extremely low cross talk between cores, gave the advantage of a very large memory effect as described in Sec. 2.2. We used the system of galvanometric scan mirrors in conjunction with an SLM (cf., Sec. 3.2) to interface the MCF with the excitation laser. Using a subset of 91 cores out of the 475 cores of the MCF, we were able to demonstrate imaging of a USAF test chart as well as 3-D control over the focal spot.⁵⁹ In addition, we demonstrated that the memory effect of our MCF allowed us to use the galvanometric scan mirrors, located at the proximal end of the MCF to scan the focal point across the sample located at the distal end of the MCF. We were able to acquire transmission images of the USAF test chart at a frame rate up to 12 Hz, limited only by the resonance frequency of the galvanometric scan mirrors. By extension, this paper demonstrated that arbitrary scan rates and also arbitrary scan patterns can be achieved in lensless endoscopes without resorting to complex calculations of the phase mask on the SLM, using instead the memory effect together with simple scan mirrors.

More recently, the work of Stasio et al.⁶¹ has also highlighted the use of the memory effect for point-scanning in a lensless endoscope. The imaging fiber bundle employed by the authors had a significant amount of cross talk between cores, which led to a decreased memory effect. As a result, the reduced memory effect only permitted scanning of a focal spot over a range of some micrometers which, nevertheless, could still significantly reduce the complexity of the phase calibration step, which the authors performed by digital phase conjugation.

4.3.

Two-Photon Imaging

From early on, a major aim of our work on lensless endoscopes was to reach a concept that would allow for performing two-photon imaging⁵ in a lensless endoscope. Two-photon imaging puts entirely new requirements on the waveguide that is to be employed in the lensless endoscope. The reason being that two-photon fluorescence is a nonlinear process that has to be excited by ultrashort laser pulses of 100 fs duration in order to be efficient. When sending ultrashort pulses through a waveguide, two new potential signal impairments must be considered, which are not encountered when using continuous lasers: temporal broadening by chromatic dispersion and temporal broadening by modal dispersion. As discussed in Sec. 2.3, MCFs with single-mode cores are well suited for the task. In Ref. 45, we demonstrated for the first time two-photon imaging acquired with true endoscopic detection. We might take this occasion to also add that MCFs are only very weakly subject to impairments caused by nonlinear effects in the MCF itself, the reason being that the power is distributed over many cores, ensuring that intensity inside the MCF is never high enough to induce nonlinear effects. Employing the MCF in Fig. 2(b), we were able to acquire two-photon images of a test sample, a sample of rhodamine 6G crystals. Thanks to the addition of the double cladding to the MCF compared to the previous design, backscattered two-photon fluorescent signal could be collected by the MCF itself and transported back to a detector located at the proximal side. Signal collection through the MCF was quite efficient too—when comparing distal and proximal signal detection we found hardly any difference.

While our approach relies on having an MCF that is optimized for transport of ultrashort pulses, a quite different approach to two-photon imaging in lensless endoscopes is represented by the recent work by Conkey et al.,⁶² which, in effect, describes an approach where one can use a commercial imaging fiber bundle and partially compensate for its imperfections with coherence-gated digital phase conjugation.

4.4.

Compensation of Residual Group Delay

The experiments on the two-photon lensless endoscope quickly revealed that the major source of impairment of the excitation pulses in the MCF was due to “inter-core group delay dispersion,” i.e., the fact that the fundamental modes in the single-mode cores of the MCF have slightly different propagation constants due to small inhomogeneities and, thus, travel at slightly different velocities. This leads to spread in the group delays of the pulses emitted from the cores, even though they were injected with zero relative delay. Exemplifying, an effective index mismatch between two cores of only $3\times {10}^{-5}$ results in a group delay spread between the two cores of 150 fs over 1 m of MCF, which is similar to the pulse width of common ultrafast lasers. The negative impact of group delay spread can be estimated by simple calculations,⁴⁶ if the standard deviation of the group delay spread is one pulse width, two-photon yield is 7.5 times decreased compared to the ideal case.

In Ref. 46, we presented a new experimental concept capable of partially compensating the group delay spread in an MCF. The employed wave front shaping scheme was of the two-SLM variety, cf., Sec. 3.3, the first SLM (SLM1) was a segmented DM, whereas the second SLM (SLM2) was a traditional liquid-crystal-based SLM. SLM1 is divided into hexagonal segments, one per core in the MCF, and on each segment $i$ can be inscribed a linear phase gradient that changes the direction of beamlet $i$ by $\mathrm{\Delta }{\overrightarrow{k}}_i$. SLM2 is similarly divided into hexagonal segments, and segment $i$ on SLM1 is imaged onto a segment $i$ on SLM2. Onto segment $i$ on SLM2 is inscribed a linear phase ramp, which once again changes the incident beamlet direction by $\mathrm{\Delta }{\overrightarrow{k}}_i$. Note that in this configuration, all beamlets reflected off SLM1 with a certain $\mathrm{\Delta }\overrightarrow{k}$ meet at the same point in the Fourier plane. In addition, all beamlets, whatever their $\mathrm{\Delta }\overrightarrow{k}$, are reflected from SLM2 parallel to one another. Put differently, we use the tip/tilt degree of freedom on SLM1 to divide the beamlets into subgroups, each destined to traverse a zone of a certain thickness of a compensation plate located in the Fourier plane. The setup can be dynamic—thanks to the tip/tilt degree of freedom the members of the subgroups can be interchanged on the fly. Figure 3(c) shows a schematic view of this concept.

Using phase-shifting spectral interferometry, we mapped out the group delay spread of a 30-cm long piece of the MCF in Fig. 2(b) and found the group delays to be distributed with a standard deviation of 123 fs. We then applied phase and group delay compensation using the two-SLM wave front shaping scheme with which we could narrow the standard deviation 65 fs. In order to verify the performance gain using this compensation scheme, we established a focus $Z=1000\mu \mathrm{m}$ from the distal tip of the MCF. The result can be seen in Figs. 4(a) and 4(b) in the case where both phase and group delay are compensated and in Figs. 4(c) and 4(d) in the case where only phase is compensated. It can be appreciated that the focus intensity is 30% higher in the former case.

Fig. 4

Compensation of residual group delay in an MCF for lensless endoscopy. (Bottom row) Images of distal focus and (upper row) horizontal cuts. (a) and (b) With group delay compensation. (c) and (d) Without group delay compensation. Scale bar, $50\mu \mathrm{m}$.

An alternative approach to characterizing the core-to-core variations in propagation constants of an MCF, based on a Markov chain Monte Carlo process, is described by Mosley et al.⁶⁷

Strictly speaking, it is only with this newfound ability to control individual group delays in an MCF that one can claim to have achieved complete control over the wave front emanating from an MCF; all varieties of lensless endoscopes exert control over the zeroth-order phase (the phase), the concept described here additionally exerts control over the first-order phase (the group delay), and, cf., Sec. 2.3, the second- and higher-order of the phase (group delay dispersion and higher-order dispersion) can be compensated globally.

4.5.

Field-of-View Extension with Aperiodic MCF

In a recent paper,⁴⁷ we experimentally demonstrated the added capabilities brought about by the most recent MCF design, an “aperiodic MCF” with its cores located at random positions [Fig. 2(c)]. The design of the aperiodic MCF was born out of the necessity to eliminate a stubborn imaging artifact that is ever-present when using the periodic MCFs Figs. 2(a) and 2(b). This artifact, and its solution, can be understood by calculating the relation between core positions and excitation intensity distribution on the sample. Even though lensless endoscopes based on MCF typically operate in the Fresnel regime, a good approximation can be obtained by assuming that the sample is located in the far-field. In this case, the field on the sample (far-field) can be found by Fourier transform of the field in the MCF distal endface (near-field). In the general case, the cores are located at ($x_i$, $y_i$) and the core positions can be defined in an “array factor” (AF)

In an MCF, all the cores emit with the same intensity distribution. In this example, we additionally suppose that all the emissions are in-phase, then the fields of all the cores are also identical $E_{1\text{core}}$. The general expression for the near-field is then

Fig. 5

Imaging properties in lensless endoscopes with different types of MCF. (a) Imaging of a USAF test chart located at $Z=1000\mu \mathrm{m}$ using the periodic MCF in Fig. 2(b). Scale bar, expected period of recurrence of replica peaks due to periodicity of the $\mathrm{MCF}=74\mu \mathrm{m}$. (b) Imaging of a USAF test chart located at $Z=600\mu \mathrm{m}$ using the aperiodic MCF in Fig. 2(c). Scale bar, expected period of recurrence of replica peaks due to quasi-periodicity of the $\mathrm{MCF}=35\mu \mathrm{m}$.

It is clear from this expression that one expects an aperiodic MCF to result in a single-peaked PSF, which in turn should permit to acquire an artifact-free image over the entire attainable field-of-view given by the width of ${\tilde{E}}_{1\text{core}}$. In Ref. 47, we designed and fabricated the aperiodic MCF shown in Fig. 2(c) and we integrated it in a lensless endoscope to confirm the added imaging capabilities brought about by this aperiodic MCF. A resulting image of a USAF test chart is shown in Fig. 5(b). One readily appreciates the absence of duplicate images and the ability to image over the entire attainable field-of-view, which represents a radical improvement over the case of periodic MCF Fig. 5(a). So far, we have done this demonstration only with a single-cladding variety of the aperiodic MCF [Fig. 2(c)], which meant that we had to detect sample signal in the forward direction, rather than through the MCF itself. However, we have recently been able to realize an aperiodic MCF with a double cladding, which can be shown in Fig. 2(d), and will permit us to combine true endoscopic imaging with artifact-free imaging over the entire available field-of-view in a lensless endoscope based on aperiodic MCF.

5.1.

MCF Bending Compensation

It must be noted that in all of our previous work cited here we have not accounted for extrinsic factors, notably bending of the MCF. We expect stress-induced changes in the refractive index due to bending to play a significant role. In the application of lensless endoscopes beyond a few special cases, an active compensation for bending is essential. This remains one of the key challenges toward the operation of robust lensless endoscopes in a clinical setting and represents the next logical step in the progression of our work. We would like to point out that the wave front shaping concepts we have reviewed here would allow for a fast compensation of phase and group delays introduced by bending, provided it can be measured in real time.

A potential path toward real-time bending compensation has been reported by Caravaca-Aguirre et al.⁶⁸ and Farahi et al.⁶⁹ using digital phase conjugation together with a “guide star” located at a distal point ($X,Y,Z$). The lensless endoscopes employed MMFs as the waveguide and since MMFs display zero memory effect (cf., Sec. 2.2) the bending compensation is only applicable to the ($X,Y,Z$) given by the guide star. This bending compensation has not yet been demonstrated with MCF, but if it could be demonstrated, the infinite memory effect of MCF would make sure that bending compensation is valid for all ($X,Y,Z$) within the attainable field-of-view delimited by the NA of the single-mode cores.

Another potential solution could be the one reported in Ref. 58. In an MCF-based lensless endoscope, the authors achieved a dynamic compensation of phase perturbations at 100 Hz. This was enabled by a real-time measurement of the phases of individual cores from an interferogram. This method, however, requires to mix the light emitted from the MCF with a reference beam at the distal end, which is incompatible with most practical applications. Finally, we might mention that, as demonstrated by Plöschner et al.,³⁹ it is possible to calculate to very good precision the transmission matrix of an MMF, provided its characteristics and its geometrical configuration are known; such an approach—although computationally heavy at the moment—might also be a route toward real-time bending compensation.

5.2.

Polarization Control

As already discussed in Sec. 2.5, the cores of MCFs with nonbirefringent cores act as randomly oriented waveplates, so about half of the light emitted by the MCF will be in either polarization state. With such an MCF in a lensless endoscope—whose concept relies on the interference between emitted beamlets—one would then require a wave front shaping scheme that controls horizontal and vertical polarization states to make both states interfere constructively on the same ($X,Y,Z$).

While the wave front shaping methods that we have discussed in this paper (Sec. 3) are all single-polarization techniques (due to the liquid-crystal SLM being a polarizing device), recently, several papers have employed wave front shaping methods that can exert control over both polarization states of light, see Refs. 39 or 70. Such a dual-polarization wave front shaping scheme has not yet been utilized together with an MCF. Nevertheless, it seems clear that future implementations of MCF-based lensless endoscopes should integrate some form of polarization control to limit excess losses. Birefringent, polarization-maintaining MCFs have been demonstrated, which do not require dual-polarization wave front shaping to minimize excess loss.^57,58 However, birefringent MCFs are significantly more complex to fabricate than nonbirefringent MCFs, and it remains to be seen whether a lensless endoscope using dual-polarization wave front shaping with a nonbirefringent MCF will prove to be a more viable solution.

In any case, polarization control in a nonbirefringent MCF would add a new layer of complexity to the phase calibration step (Sec. 3.4), since two sets of ${\delta }_i$ would have to be measured, one for each polarization state. In Ref. 54, we described and tested a method of single-shot polarimetry and applied it to the characterization of two different types of MCF. The method is real time and can simultaneously retrieve the polarization states of the light emitted by a large number of cores and hence recover the Jones matrix of each core. This method cannot replace the phase calibration step (Sec. 3.4), since it provides no information on the relative phases, but it might be used as a supplement to the phase calibration step to facilitate a dual-polarization phase calibration of an MCF.

5.3.

Imaging Speed

At the moment, the performance of the presented two-photon lensless endoscopes does not equal that of other recently demonstrated two-photon endoscopes, e.g., the fiber-scanning two-photon endoscope reported by Brown et al.,⁷¹ where frames were acquired in vivo at 4.1 Hz with 75 mW in a $1\text{-}\mu \mathrm{m}$ spot. In Ref. 45, we obtained only 2 mW in a $4\text{-}\mu \mathrm{m}$ spot and, thus, did not have sufficient excitation intensity to achieve high framerate in two-photon imaging. In Ref. 44, we, however, demonstrated video-rate imaging, but we have so far not been able to combine fast frame rate with two-photon contrast.

One component in the equation to reach higher imaging speed is the excitation intensity; as a general fact, signal yield in multiphoton processes increases inversely with the size of the focus. Lensless endoscopes, and particularly the MCFs they are based on, should therefore in the future be optimized to allow for smaller foci. This is conjoint with another issue that of spatial resolution and will be discussed in Sec. 5.4.

Another component is the optical components that facilitate the point scanning. Liquid-crystal-based SLMs typically allow update rates at some 10 s of Hz, although variants exist that allow updates rates up to 100 s of Hz. DMs, being mechanical devices, can reach significantly higher update rates of several kilohertz, or even several 10s of kilohertz. However, imaging speed and resolution requirements for quality in vivo imaging are on the order of 4 Hz frame rate and $100\times 100$ pixels per image. This would require point-scanning at a rate of 40 kHz, a rate that liquid-crystal-based SLMs and most DMs cannot reach. So, it would seem that in order to satisfy these imaging requirements, it is mandatory for lensless endoscopes to take advantage of the memory effect in MCFs, which permit point-scanning aided by traditional beam-scanning elements, like pairs of galvanometric scan mirrors (cf., Sec. 3.2).

5.4.

Mode Density

Under quite broad assumptions, the mode density, or simply the fill factor for MCF with single-mode cores, is what determines the Strehl ratio, i.e., the fraction of energy in the distal focus to total energy. Also, the total number of modes is what determines how many independent pixels can be obtained in an image.

The mode density (core density) of the MCF in Fig. 2(b) is around $1\times {10}^{-3}{\mu \mathrm{m}}^{-2}$, whereas the mode density in a typical MMF (core diameter $62.5\mu \mathrm{m}$, NA 0.3) is around $0.1{\mu \mathrm{m}}^{-2}$. Considering this factor 100 difference, clearly, this is the point where MCF-based lensless endoscopes are at a huge disadvantage compared to MMF-based lensless endoscopes. It will be important for the future of MCF-based lensless endoscopes to either devise MCF with increased core density or to otherwise find usable workarounds.

Several examples from the literature may indicate the way forward. In Refs. 72 and 73, an MCF with extremely closely spaced cores ($\mathrm{\Lambda }=1.5\mu \mathrm{m}$) and a core density of around $0.4{\mu \mathrm{m}}^{-2}$. Such an MCF effectively closes the gap in mode density between MCF and MMF. However, as the modal analysis showed, an MCF with closely spaced cores does not support localized modes confined to individual cores but rather supermodes delocalized over a large area, much like the eigenmodes of an MMF. Such an MCF with dense core packing would lose the memory effect advantage of low-coupling MCF. In Refs. 74 and 75, a potential solution for increasing the number of independent pixels in the lensless endoscope image is reported. By placing a scattering medium at the distal tip of the endoscope fiber (an MMF, in this case), a speckle field is generated with all $k$-vector values; this permits us to generate smaller focus spots over the entire field-of-view and, thus, a higher pixel count. But the price to pay is that the Strehl ratio is degraded, the ratio of energy in the focused spot decreases and a broad background appears. This would be a nuisance in one-photon imaging, but the solution can be viable for two-photon image contrast which discriminates against the weak background. Reference 76 shows results along the same lines, it showed that a short piece of MMF can act in a way similar to the scattering medium cited just before. The big difference being that an MMF does not scatter light in all directions but constrains light to a certain angular cone. Using an MMF rather than a scattering medium to increase the number of effective pixels should, therefore, give a better (or at least more controllable) tradeoff between pixel count increase and loss of Strehl ratio. In addition, Ref. 76 highlighted that the mode scrambling effect is effective even in short lengths of MMF, which are so short that modal dispersion is not detrimental.

5.5.

Nonlinear Contrast Mechanisms

The proof of principle two-photon lensless endoscope reported by us and others^44,58,62 opens the door to many other nonlinear image contrasts in lensless endoscopes. Any kind of nonlinear microscopy that can proceed by point scanning and relies on a single-excitation beam, e.g., three-photon fluorescence, second-harmonic generation, and third-harmonic generation microscopy, could be implemented in a lensless endoscope following the same principles.

Numerous other contrast mechanisms exit that require excitation with several excitation beams of different colors, e.g., nonlinear Raman-type processes such as coherent anti-Stokes Raman scattering and stimulated Raman scattering imaging. The challenge in fiber-based endoscopy using several different excitation beams is to avoid that nonlinear interaction between the excitation happens in the waveguide before the excitation pulses have reached the sample. MCFs offer a convenient solution to this challenge. By letting each excitation pulse propagate in a separate subset of cores, all nonlinear interaction between excitation pulses can be avoided during the transport of the excitation pulses through the MCF to the sample. Thus, each color can be controlled independently of the other, and the two can be brought to focus on the same ($X,Y,Z$) on the sample. The phase calibration step would have to be done separately for the two, but once this is done, the point scanning, active phase compensation, and so on can, in principle, be done by keeping track of the frequency ratio and compensating accordingly in the algorithms.