High performance data centers are increasingly bottlenecked by the energy and communications costs of interconnection networks. Our recent work has shown how integrated silicon photonics with comb-driven dense wavelength-division multiplexing can scale to realize Pb/s chip escape bandwidths with sub-picojoule/bit energy consumption. We use this emerging interconnect technology to introduce the concept of embedded photonics for deeply disaggregated architectures. Beyond alleviating the bandwidth/energy bottlenecks, the new architectural approach enables flexible connectivity tailored for specific applications.

A JTC-based photonic neural network accelerator architecture was recorded at SPIE Optics + Photonics held in San Diego, California, United States 2022.

The concept of Neuromorphic Photonics introduces advantages of optical information processing into the neuromorphic engineering domain. Most of the current efforts in the field are focused on identifying the potential mechanisms for useful and flexible spiking neuron implementation. We propose a new approach in which microcavities exhibiting strong exciton-photon interaction may serve as building blocks of optical spiking neurons. Our experiments prove similarities between polariton in-out pulse characteristics and the fundamental spiking behavior of a biological neuron. These effects, evidenced in photoluminescence characteristics, arise within sub-ns timescales. The presented approach provides means for energy-efficient ultrafast processing of spike-like laser pulses.

Artificial neural networks (ANNs) have become a staple computing technique in many fields. Yet, they differ from classical computing hardware by taking a connectionist and parallel approach to computing and information processing. Here, we present a high performance, scalable, fully parallel, and autonomous PNN based on a large area vertical-cavity surface-emitting laser (LA-VCSEL). We implement 300+ hardware nodes and train the network to perform up to 6-bit header recognition, XOR classification and digital to analog conversion. Moreover, we investigate the impact of different physical parameters, namely, injection wavelength, injection power, and bias current on performance, and link these parameters to the general computational measures of consistency and dimensionality.

In conventional approaches to computer-vision tasks such as object recognition, a camera digitally records a high-resolution image and an algorithm is run to extract information from the image. Alternative image-sensing schemes have been proposed that extract high-level features from a scene using optical components, filtering out irrelevant information ahead of conversion from the optical to electronic domains by an array of detectors (e.g., a CMOS image sensor). In this way, images are compressed into a low-dimensional latent space, allowing computer-vision applications to be realized with fewer detectors, fewer photons, and reduced digital post-processing, which enables low-latency processing. Optical neural networks (ONNs) offer a powerful platform for such image compression/feature extraction in the analog, optical domain. While ONNs have been successfully implemented using only linear operations, which can still be leveraged for computer-vision applications, it is well known that adding nonlinearity (a prerequisite for depth) enables neural networks to perform more complex processing. Our work realizes a multilayer ONN preprocessor for image sensing, using a commercial image intensifier as an optoelectronic, optical-to-optical nonlinear activation function. The nonlinear ONN preprocessor achieves compression ratios up to 800:1. At high compression ratios, the nonlinear ONN outperforms any linear preprocessor in terms of classification accuracy on a variety of tasks. Our experiments demonstrate ONN image sensors with incoherent light, but emerging technologies such as metasurfaces, large-scale laser arrays, and novel optoelectronic materials, will provide the means to realize a variety of multilayer ONN preprocessors that act on coherent and/or quantum light.

Understanding how cells form tissues and how organs grow into their final shape relies on direct observation of these self-organization processes in living specimen. In cell and developmental biology, such observations are made possible by recent breakthroughs in volumetric 3D live microscopy, providing tissue-level data at sub-cellular resolution. These 3D+time datasets allow us to directly observe the processes of life, but they also define a new set of computer-science challenges: How to store and process Terabyte-sized 3D images? How to learn physical principles from them? How to visualize data at rates of several Gigabytes per second? This has given rise to the field of big-data bio-image informatics. In my talk, I will give an overview of the approaches we developed over the past few years. I will present a novel image representation, the Adaptive Particle Representation, which can replace the traditional pixel grids in such applications. I will show how the APR has enabled real-time processing and visualization of very large microscopy volumes, and how one can adapt convolutional neural networks to natively operate on this representation without intermediately having to go back to pixels. Finally, I highlight some recent advancements in statistical learning theory that enable us to learn interpretable and physically consistent active matter models directly from noisy microscopy videos, closing the loop back to the initial physical hypothesis of tissue self-organization.

We show that a custom ResNet-inspired CNN architecture trained on simulated biomolecule trajectories surpasses the performance of standard algorithms in terms of tracking and determining the molecular weight and hydrodynamic radius of biomolecules in the low-kDa regime in NSM optical microscopy. We show that high accuracy and precision is retained even below the 10-kDa regime, constituting approximately an order of magnitude improvement in limit of detection compared to current state-of-the-art, enabling analysis of hitherto elusive species of biomolecules such as cytokines (~5-25 kDa) important for cancer research and the protein hormone insulin (~5.6 kDa), potentially opening up entirely new avenues of biological research.

Computational Miniature Mesoscope (CM2) is a novel fluorescence imaging device that achieves single-shot 3D imaging on a compact platform by jointly designing the optics and algorithm. However, the low axial resolution and heavy computational cost hinder its biomedical applications. Here, we demonstrate a deep learning framework, termed CM2Net, to perform fast and reliable 3D reconstruction. Specifically, the multi-stage CM2Net is trained on synthetic data with realistic field varying aberrations based on a 3D linear shift variant model. We experimentally demonstrate that the CM2Net can provide 10x improvement in axial resolution and 1400x faster reconstruction speed.

We report the all-optical and twin-image-free reconstruction of inline holograms using diffractive networks. Our numerical results reveal that these diffractive network designs, when properly trained using error back-propagation algorithms, generalize very well to reconstruct new, unseen holograms at the speed of light propagation, without any external power source (except the illumination light). These diffractive hologram reconstruction networks also exhibit improved power efficiency and extended depth-of-field. With their passive operation and orders-of-magnitude faster reconstruction speed than digital hologram reconstruction systems, diffractive networks can find numerous applications in holographic imaging and display-related applications.

We report a computer-free method to image through random, new diffusers at the speed of light using passive diffractive optical networks composed of spatially-engineered transmissive layers. These diffractive layers were designed using deep learning in a computer with image pairs containing diffuser distorted optical fields and the corresponding distortion-free images (ground truth). After this one-time training effort, the resulting diffractive layers were fabricated to form a physical network to all-optically reconstruct unknown objects through random, unknown diffusers, without requiring any power except for illumination. This diffractive computational imager might find applications in various fields, e.g., atmospheric sciences, biomedical imaging, defense/security.

This work introduces MAGIK, a geometric deep learning framework for characterizing dynamic properties from time-lapse microscopy. MAGIK exploits geometric deep learning capability to capture the full spatiotemporal complexity of biological experiments using Graph Attention Networks. By processing object features with geometric priors, the neural network is capable of performing multiple tasks, from linking coordinates into trajectories to inferring local and global dynamic properties of the biological system. We demonstrate the flexibility and reliability of MAGIK by applying it to real and simulated data corresponding to a broad range of biological experiments.

Deep learning-based microscopic imaging methods commonly have limited generalization to new types of samples, requiring diverse training image data. Here we report a few-shot transfer learning framework for hologram reconstruction that can rapidly generalize to new types of samples using only small amounts of training data. The effectiveness of this method was validated on small image datasets of prostate and salivary gland tissue sections unseen by the network before. Compared to baseline models trained from scratch, our approach achieved ~2.5-fold convergence speed acceleration, ~20% training time reduction per epoch, and improved image reconstruction quality.

Intracavity optical tweezers have been proven successful for trapping microscopic particles at very low average power intensity – much lower than the one in standard optical tweezers. This feature makes them particularly promising for the study of biological samples. The modeling of such systems, though, requires time-consuming numerical simulations that affect its usability and predictive power. With the help of machine learning, we can overcome the numerical bottleneck – the calculation of optical forces, torques, and losses – reproduce the results in the literature and generalize to the case of counterpropagating-beams intracavity optical trapping.

Quantitative phase imaging (QPI) has emerged as a powerful, label-free tool for revealing specimens' optical path length information. We demonstrate diffractive QPI networks that all-optically synthesize the quantitative phase image of an object by converting the optical path length variations at the input into spatial intensity distributions at the output plane. These diffractive QPI networks are spatially-engineered using deep learning to form all-optical coherent processors, and can enable power-efficient, high frame-rate and compact quantitative phase imaging systems that will be useful for various applications, including, e.g., on-chip microscopy and sensing.

We present LodeSTAR, a label-free, single-shot particle tracker. We design a method for exploiting the symmetries of problem statements to train neural networks using extremely small datasets and without ground truth. We demonstrate that LodeSTAR outperforms traditional methods in terms of accuracy and that it reliably tracks experimental data of packed cells. Finally, we show that LodeSTAR can exploit additional symmetries to extend the measurable particle properties to the axial position of objects and particle polarizability.

Neural network training and validation rely on the availability of large high-quality datasets. However, in many cases only incomplete datasets are available, particularly in health care applications, where each patient typically undergoes different clinical procedures or can drop out of a study. Here, we introduce GapNet, an alternative deep-learning training approach that can use highly incomplete datasets without overfitting or introducing artefacts. Using two highly incomplete real-world medical datasets, we show that GapNet improves the identification of patients with underlying Alzheimer's disease pathology and of patients at risk of hospitalization due to Covid-19. Compared to commonly used imputation methods, this improvement suggests that GapNet can become a general tool to handle incomplete medical datasets.

Reflectance confocal microscopy (RCM) can provide in vivo images of the skin with cellular-level resolution; however, RCM images are grayscale, lack nuclear features and have a low correlation with histology. We present a deep learning-based virtual staining method to perform non-invasive virtual histology of the skin based on in vivo, label-free RCM images. This virtual histology framework revealed successful inference for various skin conditions, such as basal cell carcinoma, also covering distinct skin layers, including epidermis and dermal-epidermal junction. This method can pave the way for faster and more accurate diagnosis of malignant skin neoplasms while reducing unnecessary biopsies.

Immunohistochemical (IHC) staining of the human epidermal growth factor receptor 2 (HER2) is routinely performed on breast cancer cases to guide immunotherapies and help predict the prognosis of breast tumors. We present a label-free virtual HER2 staining method enabled by deep learning as an alternative digital staining method. Our blinded, quantitative analysis based on three board-certified breast pathologists revealed that evaluating HER2 scores based on virtually-stained HER2 whole slide images (WSIs) is as accurate as standard IHC-stained WSIs. This virtual HER2 staining can be extended to other IHC biomarkers to significantly improve disease diagnostics and prognostics.

Label-free characterization of biological matter across scales was recorded at SPIE Optics + Photonics held in San Diego, California, United States 2022.

Histological staining is an indispensable tool for both biomedical research and clinical diagnosis of various diseases. However, current practices of histological staining often involve high-cost and laborious procedures with non-negligible environmental impact. Here we present a virtual histological staining framework that digitally stains unlabeled human tissue sections using autofluorescence microscopy and deep learning techniques. By training deep neural network models, we digitally replicated multiple stains on unlabeled tissue sections and accurately matched the images of the same samples after being histochemically stained in pathology labs. The success of our framework was demonstrated by predicting H&E stain, special stains, immunohistochemical stains, and immunofluorescence stains on multiple types of tissue sections, including lung, brain, breast, kidney, glands, etc. Beyond visual comparisons, blind studies led by board-certified pathologists further confirmed the equivalent staining quality and diagnostic values of the virtually stained images compared against their histochemically stained counterparts. By eliminating the variability introduced by the technician, reagents, environmental conditions, and digitization during histological staining, our virtual staining method produces consistent staining results across different sample slides, providing an ideal entry point for the ever-growing computer-aided pathological image analysis pipelines. More importantly, this non-destructive staining method allows the prediction of multiple stains on the same tissue section, promotes accurate evaluation of the exact same biological contents under multiple stains, which also helps reduce the sample volume that needs to be excised from the patients. To conclude, the presented virtual staining framework provides a label-free, high-quality, cost-effective, and eco-conscious method to the histological staining field.

Microfluidics is commonly ruled by pressure driven flows enabling the transport of material on large scales incorporating different kinds of functionality for sensing flow control or chemical synthesis. Yet, a local control of fluids and dissolved species is difficult due to the macroscopic nature of the exerted pressure gradients. Here we present our efforts to control liquids and dissolved species at the microscale using thermo-fluidic approaches. We employ optically controlled thermo-osmotic, thermophoretic, and thermoviscous flows to induce fluid flow to sense, localise, or separate different species in solution. We introduce different spectroscopic and microscopic signals to report on the local properties and composition of the solution with the help of machine learning approaches to track and classify species in real time to provide a feedback to steer the system into desired directions.

We present a technique to track microplanktons through generations, and continuously measure their three-dimensional position and dry mass. By combining holographic microscopy with deep learning, the technique is minimally invasive and non-destructive for plankton cells, allowing quantitative assessments of trophic interactions such as feeding events, biomass increase throughout the cell cycle. We evaluate the performance of the method, by applying it to various plankton species belonging to different trophic levels. Finally, we demonstrate the dry mass transfer from cell to cell in prey-predator interactions, and show the growth dynamics from division to division in diatoms.

Minimally invasive endoscopes are indispensable in biomedicine. Coherent fiber bundles (CFB) enable lensless endoscopes. However, the aberration correction is challenging. Instead of involving bulky devices, deep neural networks (DNN) will be used. The novel approach uses speckles, which are decoded by DNN to retrieve the 3D object information. Besides this far-field approach, near-field CFB-based high-resolution imaging is promising for neurosurgery. However, the inherent honeycomb artifacts of CFB reduce the resolution. The inherent artifact is eliminated by DNN and high frequency information could be retrieved. Both methods have smart concepts in common, and both pave the way towards early recognition of diseases.

Implementation of graph theory for novel approaches to analyze the brain in an easy, accurate, and reproducible manner requires a modern solution tool. Here, we present BRAPH 2.0 (BRain Analysis using graPH theory version 2.0, www.braph.org), a comprehensive extension of the first version of this software that includes these novel approaches. The MatLab-based BRAPH 2.0 uses object-oriented programming and a completely new software engine to provide clear, robust, clean, modular, maintainable, and testable code. The core of BRAPH 2.0 consists of a set of functions that can automatically transform a user-provided script into an object that is intertwined with the rest of the code. In this way, BRAPH 2.0 provides a scaffold on which users can define custom analysis pipelines with alternative network measures, additional statistical tests, or different options for network visualization.

In this study we assess the dynamic modular organization in patients at different stages of Alzheimer’s Disease (AD) using resting-state functional MRI data. We built a temporal multiplex network using the time-series of 200 regions from different non-overlapping time windows. The organization of these networks was evaluated using the temporal multilayer modularity and node flexibility. We found changes in the temporal dynamics of functional brain networks in AD such a loss of flexibility as well as rearrangement and loss of brain modules across time windows. These alterations were more prominent in cognitively impaired groups, suggesting they might be useful in characterizing clinical progression in later disease stages.

There have been conflicting results regarding the differences between men and women on the function and structure of the brain. To address this question, we propose a novel method to distinguish the effects of sex on brain structure and function based on identifying subgroups that maximize the between-sex differences. In a large sample of 19975 women and 17568 men, we demonstrate that our method can identify individuals at the extremities of the "maleness-femaleness" continuum and is able to quantify the maleness/femaleness of their features. These findings have widespread implications for studies assessing sex and its impact on the brain.

Functional heterogeneity in the posterior medial parietal cortex is associated with cognition was recorded at SPIE Optics + Photonics held in San Diego, California, United States 2022.

A novel self-supervised deep learning (DL) method is developed to compute personalized brain functional networks (FNs) for characterizing brain functional neuroanatomy based on functional MRI (fMRI). Specifically, a DL model of convolutional neural networks with an encoder-decoder architecture is developed to compute personalized FNs directly from fMRI data. The DL model is trained to optimize functional homogeneity of the personalized FNs without utilizing any external supervision in an end-to-end fashion. We demonstrate that a DL model trained on fMRI scans from the Human Connectome Project can identify personalized FNs and generalizes well across four different datasets. We further demonstrate that the identified personalized FNs are informative for predicting individual differences in behavior, brain development, and schizophrenia status. Taken together, the self-supervised DL facilities rapid, generalizable computation of personalized FNs.

Deviations from the law of Brownian motion, typically referred to as anomalous diffusion, are ubiquitous in science and associated with non-equilibrium phenomena, flows of energy and information, and transport in living systems. In the last years, the booming of machine learning has boosted the development of new methods to detect and characterize anomalous diffusion from individual trajectories, going beyond classical calculations based on the mean squared displacement. We thus designed the AnDi challenge, an open community effort to objectively assess the performance of conventional and novel methods. We developed a python library for generating simulated datasets according to the most popular theoretical models of diffusion. We evaluated 16 methods over 3 different tasks and 3 different dimensions, involving anomalous exponent inference, model classification, and trajectory segmentation. Our analysis provides the first assessment of methods for anomalous diffusion in a variety of realistic conditions of trajectory length and noise. Furthermore, we compared the prediction provided by these methods for several experimental datasets. The results of this study further highlight the role that anomalous diffusion has in defining the biological function while revealing insight into the current state of the field and providing a benchmark for future developers.

Many systems in biology, physics, and finance exhibit anomalous diffusion dynamics where the mean squared displacement grows with an exponent that deviates from one. When studying time series recording the evolution of these systems, it is crucial to precisely measure the anomalous exponent and confidently identify the mechanisms responsible for anomalous diffusion. These tasks are difficult when only few short trajectories are available, a common scenario in non-equilibrium and living systems. We show that long short-time memory (LSTM) recurrent neural networks excel at characterizing anomalous diffusion from a single short trajectory. The method we developed generalizes to experimental data obtained from subdiffusive colloids trapped in speckle light fields and superdiffusive microswimmers. We discuss the performance of the method in comparison to alternative ones in the context of the Anomalous Diffusion Challenge. In closing, we address the interpretability of the method.

Machine learning is emerging as an essential tool in many science and engineering domains, fueled by extraordinarily powerful computers as well as advanced instruments capable of collecting high-resolution and high-dimensional experimental data. However, using off-the-shelf machine learning methods for analyzing scientific and engineering data fails to leverage our vast, collective (albeit partial) understanding of the underlying physical phenomenon or models of sensor systems. Reconstructing physical phenomena from indirect scientific observations is at the heart of scientific measurement and discovery, and so a pervasive challenge is to develop new methodologies capable of combining such physical models with training data to yield more rapid, accurate inferences. We will explore these ideas in the context of inverse problems and data assimilation; examples include climate forecasting, uncovering material structure and properties, and medical image reconstruction. Classical approaches to such inverse problems and data assimilation approaches have relied upon insights from optimization, signal processing, and the careful exploitation of forward models. In this talk, we will see how these insights and tools can be integrated into machine learning systems to yield novel methods with significant accuracy and computational advantages over naïve applications of machine learning.

In recent years, new forms of structured illumination microscopy (SIM) have used near-field illumination from metamaterial substrates to increase resolution improvements past 2x. We demonstrate that the forward model of SIM can be used as the loss function to optimize a neural network on a single set of diffraction-limited sub-images. We show that this physics-informed neural network (PINN) can be used with a variety of structured illumination methods such as plasmonic and metamaterial SIM to achieve resolution improvements of 3x and 4x.

Reservoir computing as a highly efficient architecture for recurrent neural networks has been implemented in a variety of ways, including anharmonic oscillators, liquid surfaces, and optical and electronic circuits. Here, we investigate whether active particle networks that mimic fundamental dynamical processes of living systems can serve as reservoirs. In particular, we realize active particle oscillators, each consisting of an immobile and an active colloidal microparticle suspended in a layer of a liquid solution. The motion of the active particles is manipulated by a feedback system using a focused laser that stimulates the particles to float in 2D by thermophoresis [1]. The active particle is programmed to be attracted to the immobile particle with a delayed response that exhibits a pitchfork bifurcation, which introduces nonlinearity and memory into the response of a single active oscillator. Using time multiplexing, the propulsion of the oscillator is selected at different times as virtual nodes of a reservoir that are coupled to an input layer. Since the motion of the active particle is affected in a nonlinear manner with a memory of its previous state, the last node state is naturally coupled to the other nodes from different iterations due to the intrinsic property of the delayed oscillator. We illustrate the performance of the reservoir consisting of multiple oscillators with different delays by the tasks of nonlinear prediction and classification. [1]. F. Martin et al. ACS Nano 15, 2, 3434-3440 (2021)

We show that time-delayed nonlinear effects observed between exciton-polariton condensates can be used to create neural networks in which information is encoded in time. We form condensates on semiconductor microcavity using optical pulses that reach the sample at different times. Strongly nonlinear effects are induced by time-dependent interactions with a long-lived excitonic reservoir. Such nonlinearities make it possible to create a nonlinear XOR logic gate that performs operations with a picosecond time scale. A neural network based on such a logic gate performs a speech recognition tasks with high accuracy.

In this presentation, I will discuss recent progress in automated experiment in electron and scanning probe microscopy, ranging from feature to physics discovery via active learning, with the emphasis on the strucutred Gaussian Processes and hypothesis learning.