We present a new lithography-free integrated photonic processor, targeting dynamic control of spatial-temporal modulations of the imaginary index on an active semiconductor platform. Leveraging its real-time reconfigurability, we aim to realize photonic neural networks with extraordinary flexibility to perform in-situ learning and training with high accuracy. Our work delivers a brand new and ultra-flexible integrated photonic paradigm for reconfigurable networking and computing, with great potential to process large, non-local datasets with high throughputs.

Extending the multi-plane light conversion (MPLC) technique, we propose and demonstrate a 3D micro-optic system capable of performing matrix/tensor multiplications. Our proposed approach, called multi-plane light processing (MPLP), is passive and utilizes all degrees of freedom of light which makes it well-suited to surpass electronic accelerators in both scalability and energy efficiency. MPLP is an all-in-one system capable of spatial mode conversion and multiplexing, wavelength demultiplexing, hybrid coupling, and optical routing. As a result, the proposed device can perform matrix/tensor multiplications in a single clock cycle with tens of GHz speed limited by the optical modulators and photodetectors’ bandwidth. We have experimentally demonstrated proof-of-concept MPLP utilizing a spatial mode modulator performing 2×2 matrix-matrix multiplication and discuss the scaling methods to enhance its computation power. We envision the proposed PTA competing with electronic accelerators for large-scale and power-efficient artificial intelligence (AI) applications.

Deep learning acceleration with integrated photonics has typically employed a circuit-centric approach with Mach-Zehnder interferometers. This requires a large spatial footprint, which has motivated the direct training of spatial refractive index distribution within a slab waveguide. Here, we demonstrate through simulations that nonlinear optical material platforms with large electro-optic coefficients can capitalize on this approach. We show that a linear device with realistic device parameters can perform 50 by 50 unitary matrix multiplications. We also performed MNIST digit classification, achieving 90.5% classification accuracy with minimal digital preprocessing. Finally, we comment on device implementation with Lithium Niobate or Barium Titanate slab waveguides.

Coupled systems with multiple interacting degrees of freedom provide a fertile ground for emergent dynamics, which is otherwise inaccessible in their solitary counterparts. Particularly, nonlinearity and non-equilibrium dynamics enable new opportunities in coupled photonic systems that are not present in their linear and equilibrium counterparts that can have profound consequences in sensing and computing. In this talk, I will overview recent experimental progress on accessing such dynamics in time-multiplexed networks of nonlinear resonators towards computing and sensing applications. I will present demonstrations of topological dissipation, non-equilibrium spectral phase transitions, topological mode-locked lasers, non-Hermitian topologically enhanced sensing, and photonic elementary cellular automata. I will also overview the progress on integrated optical parametric oscillators (OPOs) and their networks in lithium niobate (LN) nanophotonics for classical and quantum information processing applications.

Convolution Neural Networks are one of the most used networks for Machine Learning. But convolution operation is one of the most demanding operations for electronic units to perform. Here, we present the first integrated PhotoFourier chip, capable of performing Joint Correlation Transformation on a Silicon Photonic chip, reducing the computation complexity from O(N^2) to O(N) at GHz speed rate.

Optical approaches are extremely promising for high-speed, scalable computing, which is necessary for modern deep learning and AI applications. In this study, we introduce a multilayer optoelectronic computing framework that alternates between optical and optoelectronic layers to implement matrix-vector multiplications and rectified linear functions, respectively. The system is designed to be real-time and is highly parallel, utilizing arrays of light emitters and detectors connected with independent analog electronics. We experimentally demonstrate the operation of our system and demonstrate that it outperforms a single-layer analog through simulations.

Our advanced computer vision system allows for the precise tracking of serial numbers on steel billets in challenging industrial settings. It combines cutting-edge hardware and machine learning, excelling in character recognition (99.8%) and localization while adapting to dynamic ambient lighting conditions (104 ). Moreover, it accurately measures crucial geometric parameters such as side sizes, bulging, and skewness. This multifaceted technology promises to elevate material tracking, quality assessment, and production optimization in the steel industry to unprecedented levels.

Deep learning has emerged as a powerful tool for solving complex problems in a wide range of domains. The success of deep learning can be attributed to several factors, including the availability of massive datasets, the increasing computing power of modern hardware, and the development of efficient algorithms. Still, In the modern era of information and communication technologies, the demand for faster and more efficient data transmission has driven researchers to explore novel approaches to enhance communication systems, among them is the optical approach for such a problem. In our lab, we develop a fully optical deep learning network that is based on high order spatial mode, and the ultrafast nonlinear four wave mixing interactions inside multimode fibers. We exploit the optical nonlinear interactions between waves for developing a deep learning network that is faster than any electronic based network. In this study, we present the algorithm we developed and the theoretical implementation of such network. In addition, we demonstrate our ability to decompose and classify ultrafast signals, such as temporal modes combinations, which are typically undetectable by standard devices,

We enhance the accuracy of a diffractive optical network through time-lapse-based inference, which exploits the information diversity obtained by introducing controlled or random displacements between the object and the diffractive network, relative to each other. The numerical blind testing accuracy achieved using this time-lapse-based inference scheme on CIFAR-10 images reached >62%, representing the highest accuracy achieved so far on this dataset using a single diffractive network. Beyond image classification, this framework could also open doors to broader utilization of diffractive networks in tasks involving all-optical spatiotemporal information processing, paving the way for advanced visual computing paradigms.

Coherent light-based optical computing faces challenges in implementing low-power optical nonlinearities, which are essential for optimal neural network performance. This limitation restricts the overall performance and applicability of these networks. To address this, we propose a polarization encoding scheme that enables a low-power nonlinearity. By projecting polarization to amplitude in a sinusoidal manner and subsequently converting amplitude back to the polarization domain, we can implement a non-linear diffractive neural network. The neural network utilizes polarization-encoded inputs and polarization rotation-encoded weights. Optical interconnects between neurons are achieved through the diffraction of the spatially inhomogeneously polarized wavefront. Our approach offers a low-power solution for incorporating nonlinearity into deep diffractive neural networks.

The development of microscopic technologies has enhanced the translational research between scientists, engineers, biologists, and biomedical researchers, enabling the visualization and evaluation of complex biological systems. Advances in imaging systems are essential to further develop our understanding of cellular mechanisms and apply them to new diagnostic methods and disease treatment. Over the last decade, machine learning has been heavily used in microscopy image analysis, from the classification of cells to the reconstruction of real-time images, empowering the toolbox of automated microscopy. In this invited contribution, we discuss the use of generative adversarial networks in quantitative phase imaging and super-resolution microscopy.

In imaging system design, computational applications and optical components are interdependent. End-to-end optimization, jointly optimizing hardware and software, is a prevalent approach. However, most optical simulators use ray optic models, which may lack real-world fidelity. We propose a differentiable wave optics model that accurately simulates light propagation. It exposes performance disparities among physical models. Integrated with unrolled FISTA and color filters, the system consistently yields clear measurements and accurate recovery. By noting the performance degradation caused by deviations from real-world physics, our wave optics model is a superior choice for end-to-end imaging system design.

The carrier-less phase-retrieval (PR) coherent detection is a potential solution for reducing the hardware complexity of coherent receivers in cost-sensitive applications, such as data-center interconnects. However, due to the lack of theoretical analysis, the optimum PR receiver architecture and its achievable performance have been revealed yet. In this work, we show the injectivity condition and the Cramér-Rao lower bound for the PR receivers. These results would be a useful tool for the system designers to optimize the receiver architectures and establish a fair comparison between them.

Energy-resolving photon counting detectors (PCDs) are being explored for non-destructive spectral x-ray imaging in medical and industrial applications, allowing quantitative material mapping not practical with conventional radiography. However, PCDs suffer inherent detector non-idealities that negatively impact image quality and quantitative accuracy. While analytical methods are being developed for material separation, we leverage machine learning techniques (e.g., principal component analysis and clustering) to increase flexibility by reducing the reliance on prior knowledge of the inspected object or detection properties. Through simulating various acquisition conditions, we evaluate the robustness of these machine learning techniques for material-specific mapping in spectral x-ray imaging.

Neural network-based surrogate partial differential equation (PDE) solvers are of great interest due to their potential to solve PDEs and related inverse problems in a fast and efficient manner. However, these end-to-end methods largely remain limited to predetermined problem sizes and fixed PDE parameters, preventing their application to practical simulation tasks that comprise heterogeneous PDE parameters and arbitrary domain sizes. We propose Specialized Neural Accelerator-Powered Domain Decomposition Methods (SNAP-DDM), which is a DDM-based approach to PDE problem solving in which an ensemble of specialized neural operators is trained to accelerate the solving of subdomain problems containing arbitrary boundary conditions and geometric parameters. We tailor SNAP-DDM to 2D electromagnetics problems and show how innovations in network architecture and loss function engineering can enable trained SNAP-DDM subdomain solvers with over 99\% accuracy. We also show how SNAP-DDM can be used to accurately solve a wide range of electromagnetics scattering problems in time scales comparable to traditional FDFD algorithms.

Convolutional layers are a critical feature of modern neural networks and require significant computational resources. Recently, researchers have developed optical accelerators as a low-energy, high-bandwidth approach for performing large-scale convolutions. Existing approaches perform convolutions on only one input channel to one or more output channels. Here we develop an optical convolution approach that simultaneously convolves multiple input channels each with their own set of convolutional kernels onto multiple output channels. Our approach uses a microlens array to redirect light from a 2D light emitter array through convolutional kernels encoded on an amplitude mask onto a camera. We experimentally test our multi-channel free space optical convolution approach and evaluate its performance using ray-tracing simulations. This work solves a major constraint of existing optical convolutional approaches, as modern convolutional networks use large numbers of input and output channels in convolutional layers.

We report deep learning-based design of diffractive all-optical processors for performing arbitrary linear transformations of optical intensity under spatially incoherent illumination. We show that a diffractive optical processor can approximate an arbitrary linear intensity transformation under spatially incoherent illumination with a negligible error if it has a sufficient number of optimizable phase-only diffractive features distributed over its diffractive surfaces. Our analysis and design framework could open up new avenues in designing incoherent imaging systems with an arbitrary set of spatially-varying point-spread functions (PSFs). Moreover, this framework can also be extended to design task-specific all-optical visual information processors under natural illumination.

As neural networks (NNs) become more capable, their computational resource requirements also increase exponentially. Optical systems can provide alternatives with higher parallelizability and lower energy consumption. However, the conventional training method, error backpropagation, is challenging to implement with these analog systems since it requires the characterization of the hardware. In contrast, the Forward-Forward Algorithm defines a local loss function for each layer and trains them sequentially without tracking the error gradient between different layers. In this study, we experimentally demonstrate the suitability of this approach for optical NNs by utilizing the multimode nonlinear propagation inside an optical fiber as a building block of the NN. Compared to the all-digital implementation, the optical NN achieves significantly higher classification accuracy while utilizing the optical system only one epoch per layer.

Optical computing offers a promising solution for achieving high-performance neural networks without relying heavily on electronic computing power. This work introduces a novel framework for implementing programmable neural networks. The framework enables both linear and nonlinear transformations at low optical power by leveraging multiple scattering and data repetition to achieve high-order nonlinearities. By combining linear optics and structural nonlinearity, it offers scalability and programmability for optical computing, particularly in artificial intelligence applications. The framework's ability to synthesize a learnable linear and nonlinear data transform bridges the gap between optical and digital neural networks.

A living animal exhibits remarkable ability to survive. It processes sensory input and takes actions to maximize the likelihood of survival. we show that an artificial agent powered by reinforcement learning can also spontaneously develop sensory apparatus. It can build its own bridge to connect the digital world to the physical one. This capability could be used to develop resilient agents that are adaptive in changing environments.

Holographic displays are a promising technology for AR/VR which could enable compact form-factor headsets with accurate focal cues, prescription correction, and view-dependent effects. I'll discuss current challenges in holographic displays and research directions to address them.

The arbitrary dynamic control of both amplitude and polarization distributions is attracting strong interest in laser processing field to manage the quality and to collect valuable polarization characteristics of processing materials in smart manufacturing. We present a holographic method to generate arbitrary polarization state of multiple beams by synchronizing two phase-only liquid crystal spatial light modulators (SLMs) with imaging feedback system for hologram designing of each polarization state. This research work will help to accelerate the use of liquid crystal SLMs for high-throughput and optimized additive manufacturing.

The reservoir computing paradigm has proven effective for autonomous learning and time-series prediction. While classical reservoir computers have been extensively studied, quantum counterparts are gaining attention. Quantum reservoir computers (QRCs) offer advantages like exponential phase-space dimension scaling and entanglement as a unique resource. With advancements in semiconductor fabrication techniques for quantum-photonic systems, such as coupled-cavity arrays, QRC realization is imminent. We explore the properties and quantum advantage of QRCs based on the transverse-field Ising model. Using the benchmark of linear short-term memory capacity, we evaluate the QRC's performance in terms of entanglement and covariance dimension. Possible implementations using interconnected nanolasers as a semiconductor-based quantum-photonic neural network are discussed. [Götting et al., arXiv:2302.03595 (2023)].

We demonstrate a simple yet highly effective uncertainty quantification method for neural networks solving inverse imaging problems. We built forward-backward cycles utilizing the physical forward model and the trained network, derived the relationship of cycle consistency with respect to the robustness, uncertainty and bias of network inference, and obtained uncertainty estimators through regression analysis. An XGBoost classifier based on the uncertainty estimators was trained for out-of-distribution detection using artificial noise-injected images, and it successfully generalized to unseen real-world distribution shifts. Our method was validated on out-of-distribution detection in image deblurring and image super-resolution tasks, outperforming other deep neural network-based models.

We directly transfer optical information around arbitrarily-shaped, fully-opaque occlusions that partially or entirely block the line-of-sight between the transmitter and receiver apertures. An electronic neural network (encoder) produces an encoded phase representation of the optical information to be transmitted. Despite being obstructed by the opaque occlusion, this phase-encoded wave is decoded by a diffractive optical network at the receiver. We experimentally validated our framework in the terahertz spectrum by communicating images around different opaque occlusions using a 3D-printed diffractive decoder. This scheme can operate at any wavelength and be adopted for various applications in emerging free-space communication systems.

We present a diffractive network (D2NN) design to all-optically perform distinct transformations for different input data classes. This class-specific transformation D2NN processes the input optical field, generating the output optical field whose amplitude or intensity closely approximates the transformed/encrypted version of the input using a transformation matrix specific to the corresponding data class. The original information can be recovered only by applying the class-specific decryption keys to the corresponding class at the diffractive network's output field-of-view. The efficacy of the presented class-specific image encryption framework was validated both numerically and experimentally, tested at 1550 nm and 0.75 mm wavelengths.

We present a universal polarization transformer composed of diffractive layers and linear polarizer arrays, capable of all-optically synthesizing a large set of complex-valued polarization scattering matrices between the polarization states at different positions within its input and output fields-of-view. We numerically demonstrated that our deep learning-based design could synthesize 10,000 different spatially-encoded polarization scattering matrices within a single diffractive volume. Using wire-grid polarizers and 3D-printed diffractive layers, we also demonstrated an experimental proof-of-concept by achieving an all-optical polarization permutation operation with 16. Our innovative framework can inspire new devices with versatile polarization control capabilities in various fields.

We report an optical diffractive decoder with an electronic encoder network to facilitate the accurate transmission of optical information of interest through unknown random phase diffusers along the optical path. This hybrid electronic-optical model was trained via supervised learning, and comprises a convolutional neural network-based encoder and jointly-trained passive diffractive layers. After their joint-training using deep learning, our hybrid model can accurately transfer optical information even in the presence of unknown phase diffusers, generalizing to new random diffusers never seen before. We experimentally validated this framework using a 3D-printed diffractive network, axially spanning <70λ, where λ=0.75mm is the illumination wavelength.

We introduce a unidirectional imager that facilitates polarization-insensitive and broadband operation using isotropic, linear materials. This design comprises diffractive layers with hundreds of thousands of learnable phase features, trained using deep learning to enable power-efficient, high-fidelity imaging in the forward direction (A-to-B), while simultaneously inhibiting optical transmission and image formation in the reverse direction (B-to-A). We experimentally tested our designs using terahertz radiation, providing a good match with our simulations. Furthermore, we demonstrated a wavelength-selective unidirectional imager that performs unidirectional imaging along A-to-B at a predetermined wavelength, while at a second wavelength, the unidirectional operation switches from B-to-A.

Diffractive deep neural networks utilize successive, spatially-engineered diffractive surfaces trained via deep learning to all-optically process input optical fields based on a desired transformation. We present the design of a broadband diffractive network that can all-optically perform a large set of arbitrary complex-valued linear transformations, wherein the input/output data are encoded at W different wavelength channels, each assigned to a unique linear transformation, covering, e.g., W>100-2000. This broadband diffractive visual processor may foster the development of all-optical visual processors with substantial data bandwidth and parallel computation capabilities, creating intelligent machine vision systems for all-optical processing of multi-color or hyperspectral objects/scenes.

We report on the realization of an on-chip waveguide platform capable of creating arbitrary two-dimensional refractive index profiles in situ and in real-time. The device exhibits complex multimode dynamics which we train to perform machine learning. We tune the refractive index profile in situ using a backpropagation algorithm to perform audio and image classification with up to 50-dimensional inputs. The two-dimensional programmability is realized by sandwiching a photoconductive film and a lithium niobate slab waveguide between two flat electrodes. While applying voltage between the electrodes, we program the effective index of the waveguide by projecting different light patterns onto the photoconductive film. The effective index increases by 10^-3 in illuminated regions via the electro-optic effect, free from any measurable memory effects or cyclic degradation. In conclusion, we developed a photonics platform with versatile spatial programmability that opens new avenues for optical computing and photonic inverse-design.