Future HPC and datacenter systems are expected to contain an increasingly heterogeneous set of compute and memory resources as a strategy to preserve performance scaling in the long term. This heterogeneity, combined with the low average resource usage of today’s systems, motivate resource disaggregation to allow applications to pool and compose no more than the fine-grain resources they require. In this paper, we start by motivating resource disaggregation by observing average utilization and rate of change of memory bandwidth and latency in NERSC’s Cori. We then perform an analytical analysis that quantifies if today’s photonic links and switches would meet key metrics to minimize overhead, and what that overhead will be. Finally, we perform preliminary experiments to demonstrate that even this minimal overhead penalizes application performance in some cases. Our study motivates future work on more aggressive photonic links and switches in order to make resource disaggregation more attractive in future systems.
The explosive growth in data analytics is driving an intensely growing need for compute performance. We will review our ARPA-E PINE ENLITENED experimental testbed and simulation results to motivate performance advantages of disaggregation through the use of flexible photonic interconnect networks for distributed deep learning applications.
Photonic switches promise to have a large impact on future HPC and datacenter networks. However, even if we assume ideal zero-energy photonic switches, we cannot achieve significant system-wide benefits if we do not change the rest of the system architecture. This motivates co-design between emerging photonic switches and the rest of the system in order to adapt the system network to best make use of unique features of photonic switches, as well as tailor photonic switches to better support system-wide trends such as resource disaggregation. In this paper, we discuss the architectural impact of several properties of photonic switches. For each, we provide an overview of what system-level capabilities they enable, how they can be adapted to support ongoing trends, and what other synergistic advancements would produce a better system-wide improvement. In this way, we illustrate the potential benefit of closer collaboration between the photonic and architecture communities.
In this paper, we demonstrate the integration of a SiP switching platform to improve real-world Distributed Denial of Service (DDoS) defense systems. We demonstrate how DDoS mitigation in the optical domain can be transparent to network and application layers, allowing for reconfiguration and tuning. Additionally, we show how optical domain DoS mitigation provides significant cost reduction-with a 1/3 cost reduction-compared to traditional mitigation using electronic counterparts. Our approach is ideal for data-center deployments, and our testbed topology mirrors a standard data center set up.
High bandwidth density silicon photonic interconnects offer the potential to address the massive increase in bandwidth demands for data center traffic and high performance computing. One of the major challenges in realizing silicon photonics transceivers is the integration and packing of photonic ICs (PIC) with electronic ICs (EIC). This paper presents our version one, 2.5D integrated multi-chip module (MCM) transceiver for 4 channel wavelength division multiplexing (WDM) operation, targeting 10 Gbps per channel. We identify five key areas critical to successful integration of MCM transceivers, which we have used in developing our version two MCM transceiver: integration architecture, equivalent circuit model development, PIC to EIC interface modelling, MCM I/O design, and design for assembly.
We demonstrate an optimized silicon photonic link architecture using components from the AIM PDK that achieves an ultra-low sub-pJ/bit power consumption with an aggregate bandwidth of 480 Gb/s. At the transmitter, micro-disk modulators are cascaded along a bus waveguide to select and modulate wavelength-division multiplexed (WDM) channels. At the receiver, micro-ring resonator (MRR) filters are thermally tuned to match the corresponding disks to select from the multiplexed channels. This link architecture yields an ultra-small footprint compared to Mach-Zehnder designs, improving the system scalability and bandwidth density. Additionally, using micro-resonators to select and drop the desired wavelengths from a single bus waveguide allows for straightforward integration with a frequency comb source. The energy performance of the design is optimized through sweeping over three key parameters: (i) optical power per channel, (ii) channel count, and (iii) bitrate. These parameters are the dominant sources for the crosstalk and power penalty in the link design. We identify ideal points in the design space which minimize the energy per bit while staying below the desired bit error rate (BER) of 10-12 and maintaining a realistic aggregate bandwidth. Simulations in the Synopsys OptSim environment using the AIM PDK v2.5a models confirm the functionality of the system with a BER < 10-12, acceptable for both high performance computing (HPC) and data center (DC) applications. Furthermore, optimizing the link energy consumption in the AIM PDK provides a clear path towards low-cost and high-yield fabrication suitable for application in HPC and DC systems.
The cost and complexity of future interconnects create a significant opportunity for emerging photonic tech- nologies such as fibers and switches. These technologies should be evaluated at the system level in order to determine the most efficient way they can be used, as well as provide feedback to photonic developers to better optimize for high-level impact. In this paper, we argue for the need for a systematic methodology to extract system-level models for any emerging photonic component. We then outline our past experience with extracting architectural-level metrics from device demonstrations and conducting architectural-level evaluations. Finally, we discuss qualities for a desirable solution to this problem that requires cross-community collaboration.
This paper presents the integration of multiple silicon photonic (SiP) switches within a high-performance computing environment to enable network reconfigurability in order to achieve optimized bandwidth utilization. We demonstrate a physical testbed implementation that incorporates two fabricated SiP switches capable of switching traffic under real HPC benchmark workloads. The system uses dynamic optical bandwidth steering to match its physical network topology to the traffic characteristics of the application, and achieves up to approximately 40% reduction in application execution time of the high-performance computing benchmark. We present the detailed design of the network architecture and control plane of the system, and discuss the system performance improvements that arises from bandwidth-steering with silicon photonic-based circuit switching.
We propose a flexible, software-defined optical switching fabric for cloud data centers, enabling multi-petabit per second network capacities. Our design is based on the cyclic interconnection pattern of arrayed waveguide grating (AWG) devices, whose routing functionality is complemented with recirculation fibers. Unlike traditional optical data center network proposals that rely on two independent fabrics for supporting mice and elephants, our design enables the support of flows of various sizes and requirements using a single AWG-based fabric and yields bandwidth flexibility by integrating wavelength and subwavelength switching granularities. There are two sets of connections paths in our design: dedicated paths between each pair of AWG input and output ports, and shared paths that are set up by multiple recirculation fibers. The recirculation fibers enable the statistical multiplexing of mice. As well, they provide for flexible, on-demand circuit provisioning between input and output ports. Applying Birkhoff-von Neumann matrix decomposition on a residual traffic matrix comprising the demands that cannot be supported through the dedicated paths, we come up with a weighted sum of permutation matrices that get mapped onto the set of available recirculation fibers. The calculated coefficients determine the proportion of a timeframe that the permutation matrices are serviced by distinct fibers. The matrix decomposition requires the combined scheduling of wavelength and time domains so that the AWG can operate as an adaptive flow switching device. Enhancing the functions of our wavelength-routing design with space switching using an optical MEMS switch results in extreme network scales, spanning millions of processing cores.
Datacenter networks are not only larger but with new applications increasing the east-west traffic and the introduction of the spine leaf architecture there is an urgent need for high bandwidth, low cost, energy efficient interconnects. This paper will discuss the role integrated photonics can have in achieving datacenter requirements. We will review the state of the art and then focus on advances in optical switch fabrics and systems. The optical switch is of particular interest from the integration point of view. Current MEMS and LCOS commercial solutions are relatively large with relatively slow reconfiguration times limiting their use in packet based datacenter networks. This has driven the research and development of more highly integrated silicon photonic switch fabrics, including micro ring, Mach-Zehnder and MEMS device designs each with its own energy, bandwidth and scalability, challenges and trade-offs. Micro rings show promise for their small footprint, however they require an energy efficient means to maintain wavelength and thermal control. Latency requirements have been traditionally less stringent in datacenter networks compared to high performance computing applications, however with the increasing numbers of servers communicating within applications and the growing size of the warehouse datacenter, latency is becoming more critical. Although the transparent optical switch fabric itself has a minimal additional latency, we must also take account of any additional latency of the optically switched architecture. Proposed optically switched architectures will be reviewed.
For the first time a practical control analysis of semiconductor optical amplifier (SOA) dynamics has been conducted with implementation design considerations, based on a newly developed SOA electrical Spice model. It can be used for optimal design of SOA driver and control circuits within any existing optical network. In addition, a novel dual-control loop is designed and simulated. The two control loops overcome both fast (transient) and slow (steady-state) phenomena. Simulation results of the controlled SOA dynamics are presented, with an implementation example of an SOA high-speed optical switch driver. For optical switching the SOA driver uses a pre-impulse-step injected current as the switching signal. The developed Spice SOA electrical model is used for the predistortion optimization. The simulation results show that an ultrafast driver and control circuit seems feasible. It is also shown that the SOA can operate as a variable attenuator. A dynamic time constant of ~1–2 nsec is achievable using the predistortion scheme. The design and analysis here show that the SOA technology can be used as the switch-fabric core of a bufferless optical network with switching on packet timescales.
Optical interconnects are being considered for short link data networks as a solution enabling higher aggregate bit rates and lower power consumption. For short link length interconnects, as used in chip to chip interconnects, internal system backplanes and inter-system interconnects such as blade server backplanes, storage area networks and processing clusters, requirements are quite different to those for long distance telecommunications systems. Low power consumption, latency, and size become important criteria in addition to ultra high bandwidth. In order to achieve the projected ultra high capacity and low latency needs, we are considering optical switching fabrics. The optical switch, however, brings significant changes to the interconnect architecture in terms of how routing decisions are made and how contention resolution is managed. We discuss these issues and present our results for a multiwavelength optically switched interconnect.
The input power dynamic range (IPDR) of a semiconductor optical amplifier (SOA) is extended using a moderate power holding beam, which could be readily achieved with a single DFB laser. The associated reduction of gain with improved IPDR is studied and assessed in parallel with power penalty to explore the optimum operating powers in switching applications. Holding beam powers of less than +10 dBm facilitate IPDR enhancement to 27 dB, representing an order of magnitude improvement. The achievable gain remains sufficiently high to find applications in a number of switching and routing applications.
It is desirable for data networks to have low transmission latency. This may be achieved by exploiting the short packet lengths and the high bandwidths that can be achieved using multi-wavelength operation. Semiconductor optical amplifiers (SOAs) have been demonstrated as building blocks for optical switches and have also been shown to be well suited to the fast switching required for optical packet switching [1]. We have realised an InP based add-drop multiplexer (ADM) integrated on a single 850 μm x 850 μm chip. The bit error penalty performance has previously been shown to be below 1.2 dB for each of the operating paths through the device: add, drop and through modes at 2.5 Gbit/s data rates. Further, low penalty operation has previously been demonstrated experimentally with 4 simultaneous wavelengths [2].
It is known that the dynamic range of an SOA can limit the number of wavelengths supported and that the pattern sensitivity in SOAs increases their operating penalty [3]. We investigate the multi-wavelength operation of our ADM device and show that a power penalty of less than 0.8 dB is maintained over a 20 dB input power dynamic range. We also show a -3 dB optical bandwidth of 30 nm suitable for multi-wavelength operation of cascaded ADMs. Finally we present experimental results to show that the pattern dependent operating penalty of the ADM is reduced as the number of wavelengths of asynchronous data is increased. This result may be exploited in our proposed optical data network to produce an improved optical penalty.
This paper describes the current status of Coarse Wavelength Division Multiplexing (CWDM), and then progresses to discuss how it may evolve in networking applications in the future. As WDM can enhance not only transmission but also networking systems, the paper reports a potentially low cost WDM based access node architecture, particularly suited for routing optical data packets on nanosecond timescales. The scheme is cascadable and involves the use of a simple semiconductor optical amplifier (SAO) based add-drop switch. Preliminary results concerning the operation of the add-drop switches under multi-wavelength operation are reported.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.