2.1

Interfacing data format Standards drafted for VINYL

There are various simulation codes involved in the simulation pipeline. These codes are developed in different groups with different input/output data format definitions. In order to achieve seamless exchange of data between simulation software in simulation pipelines, a standardized formatting and hierarchical organization of simulation data is needed. At the same time, this kind of format standard can also benefit from 3rd party data visualization and analysis software that support these formats.

We choose openPMD⁴ meta data standard as the interfacing data format, therefore our workflows and results become accessible, inter–operable, and reusable to stay in line with the core concepts of FAIR Data Principles.⁵ OpenPMD stands for open particle and mesh data. It was initially developed as a metadata and data hierarchy standard for particle–in–cell (PIC) simulations of high–power laser–matter interaction at the Helmholtz–Zentrum Dresden–Rossendorf. It is currently not only limited to define PIC simulation data, but, thanks to its flexibility, also widely adopted in numerous simulation codes, visualization codes, and simulation workflow platforms by defining the domain extension in addition to the base standard. The data conforming openPMD standard can be naturally written in hierarchical file formats, such as HDF5, ADIOS1, ADIOS2⁶ and JSON. It is noteworthy that the streaming function provided by ADIOS2 is promising to speed up the computing in the workflow where a large amount of data needs to be passed from one program to another.

In VINYL, we have defined the domain extensions for coherent wavefront propagation, photon raytracing, neutron raytracing and molecular dynamics simulations to support the data exchange between the simulation modules. The extensions can be found here: https://github.com/PaNOSC-ViNYL/openPMD-standard/tree/upcoming-2.0.0.

2.2

Simulation backends

In VINYL, three existed simulation applications are considered as the backends of the API: SimEx, OASYS and McStas.

SimEx⁷ is the API of the SIMEX platform for simulation of experiments at advanced laser and X-ray light sources. It brings all aspects of typical experiments at light source infrastructures into a streamline of simulation. The simulations of photon source, light transport through optics elements in the beamline, interaction with a target or sample, scattering from the latter, photon detection, and data analysis can be done with various modules within the framework. The simulation framework has been actively used for single particle imaging studies.^{8, 9}

OASYS (OrAnge SYnchrotron Suite) is a versatile and user-friendly container of APIs dedicated to optical simulations.¹⁰ It can provide a graphical environment for modelling X-ray experiments. Shadow, SRW, XOPPY, Syned are now available in OASYS. A new wave optics-based simulation package, WISEr, which is targeted at the simulation of the focusing performance of grazing X-ray optics systems, is now under developing and is fully integrable with OASYS. By implementing the photon raytracing interfacing data format, SimEx can obtain propagated beam parameters from the raytraycing simulation results of OASYS and then conduct the photonmater interaction simulations making use of the beam parameters.

McStas^{11, 12} is a general software package for Monte Carlo neutron ray-tracing simulations. The developing of it started about 20 years ago and is still on going. McStats consists of a number of components files to describe the corresponding beam-optical components in real experiments. The experimental instrument is represented by an instrumental file containing a selection of components and their position and orientation. In order to make use of today’s cloud computing service technology and to harmonize the usage of McStas to be compatible with VINYL, a python API, McStasScript¹³ was developed. McStasScript allows a user to create instrument models and handle the returned data including plotting. Existing McStas projects can be converted to python using tools included in the McStasScript package.

3.1

X-ray Gaussian source and its propagation

This example is a quickstart example for users to get familiar with the usage of the SimEx API. An X-ray source with Gaussian distribution, both temporal and spatial, is the simplest description for starting a wavefront propagation simulation.

The source is defined by its photon energy, pulse energy and photon energy relative bandwidth as shown in Fig. 2 in one jupyter notebook cell. SimEx also provides analysis functions to visualize the temporal and spectral profiles (Fig. 3) and wavefront intensity map of a 2D slice (Fig. 4). After saving the source file in HDF5 format, the output can then be input into wave propagation modules, such as WPG¹⁴ to continue the start-to-end simulation by, for eaxmple, simulating the propagation and focusing of the X-ray beam for specific X-ray optics and geometries.

Figure 2.

The parameters of a Gaussian source resemble to an ideal hard X-ray source defined in a jupyter notebook cell.

Figure 3.

The temporal and spectral profiles of the Gaussian X-ray source plotted by the analysis functions of SimEx.

Figure 4.

The intensity map of a 2D slice from the Gaussian X-ray source plotted by the analysis functions of SimEx.

3.2

Serial crystallography diffraction at European XFEL

Serial femtosecond crystallography (SFX) is a method to determine atomic-scale structures from many individual crystals of a given sample. One prevalent problem in conventional macromolecular X-ray crystallography is the difficulty in forming high diffraction quality crystals.¹⁵ Besides that, the strong radiation damage has hindered the structure determination of nanocrystals. Making use of the extra-high brilliance and ultra-short pulse (tens of femtoseconds) of XFELs, SFX makes it possible to measure the structures of micrometer-sized and smaller protein crystals at room temperature and to potentially study biomolecular dynamics using time-resolved experimental measurements.^{16, 17} The megahertz pulse repetition rate achieved at the European XFEL¹⁸ can provide efficient measurements of the data volumes necessary for high resolution time-resolved studies in a relatively short time. By collecting a large dataset of diffraction patterns from nano/micrometer sized crystals, which are delivered by liquid jets, a number of experimental results^19–24 have shown the prospects of this method and make it a class of experiments where simulation may also help provide insight and direction.

The 1 megapixel Adaptive Gain Integrating Pixel Detector (AGIPD)²⁵ is the most commonly used diffraction detector in the SPB/SFX instrument of the European XFEL. As shown in the schematic in Fig. 5 generated by EXtra-geom,²⁶ this 1M AGIPD detector consists of 4 movable quadrants, each having 4 detector tiles with 512 × 128 pixels per tile. Each tile is further divided into 2 × 8 individual Application Specific Integrated Circuits (ASICs), each having 64 × 64 pixels of size 200 μm × 200 μm. Since the ASICs cannot physically touch each other, a gap covering the area of two ordinary pixels exists horizontally.

Figure 5.

The detector geometry of AGIPD.

ADU noise, non-functional pixels and diffraction from the medium in which crystals are suspended will all add complications to extracting the exact structure information from diffraction patterns. In this example, ideal noise-free monolithic diffraction patterns and diffraction patterns on multiple detector panels are simulated using SimEx respectively and analysed with CrystFEL²⁷ to show the effects of the gaps between detector panels on the results of crystallography analysis.

We generate 200 diffraction patterns of perfect lysozyme crystals (PDB structure 3WUL²²) for both groups (monolithic and multiple-panel detector) using the CrystFEL diffraction calculator in SimEx. The X-ray beam is ideally monochromatic with photon energy of 4.972 keV and focus diameter of 130 nm. The sample-to-detector distance is 130 mm. For the noise-free monolithic diffraction patterns (Fig. 6 (a)), we use a setup of 1000 × 1000 pixels of 220 × 220 μm² size to make it comparable to the multiple-panel diffraction patterns (Fig. 6 (b)) generated with AGIPD geometry. Random rotation of the crystal is implemented during simulation to imitate the the random orientation of the sample in the experiment.

Figure 6.

The snapshots of generated datasets of (a) monolithic diffraction patters and (b) multiple-panel diffraction patterns. Since random rotation is implemented during simulation, the two diffraction patterns look different.

The diffraction patterns are then indexed with CrystFEL’s indexamajig tool, to estimate the unit cell parameters - the three lengths and three angles which describe the repeating unit of the crystal structure. In Fig. 7, the unit cell estimates from the monolithic diffraction patterns are tightly clustered around the expected values for lysozyme. By comparison, the estimates in Fig. 8 from the multi-panel diffraction patterns show markedly different distributions: there are still clear peaks at the expected values, but there is a much wider spread of bad estimates for each parameter. This test case didn’t explore different analysis parameters which may have refined or filtered out these results.

Figure 7.

The statistics of the cell parameters estimated from the monolithic diffraction patterns: a, b and c are the length of each side of the unit cell and α, β and γ are the angles between these sides. The y-axis is the frequency of each value of the lattice parameters on the x-axis.

Figure 8.

The statistics of the cell parameters estimated from the multi-panel diffraction patterns: a, b and c are the length of each side of the unit cell and α, β and γ are the angles between these sides. The y-axis is the frequency of each value of the lattice parameters on the x-axis.

This example demonstrates the functionality of SimEx to consider the multi-panel detector geometry. The analysis results indicate that the gaps between detector panels may yield errors in the estimation of cell parameters. This may result from the information missed in the gap. Noticing that only 200 patterns are included in the dataset, one notes that a larger dataset may help reduce the errors.

3.3

Polychromatic diffraction simulation using OASYS and SimEx

As an exercise to bring the SimEx photon simulation framework and OASYS optical simulation environment together, several demo jupyter notebooks are created to show how SimEx takes the output data of OASYS complying with the openPMD wavefront domain extension defined in https://github.com/PaNOSC-ViNYL/openPMD-standard/blob/upcoming-2.0.0/EXT_PRAYTRACE.md and generates the diffraction patterns using the GAPD²⁸ diffraction calculator integrated in SimEx with the wavefront data.

In this application case, we simulate the X-ray wavefront using OASYS. Beamline ID23 at ESRF²⁹ has been used as an example. It consists of a slit, Si-111 monochromator (E = 14.2 keV) and a KB-focusing mirror system. A ray-tracing simulation using ShadowOui package has been performed. For demonstration purposes a low number of rays (10⁶) and a narrow energy bandwitdh has been used. The workflow has been demonstrated in two ways. The first is using OASYS capability to generate the underlying Python code for running the simulation made in the GUI and then running the said code in the Jupyter notebook of SimEx. The second way is to use OASYS capability to run native Python code in a GUI widget, where the beamline output is linked directly to a series of two widgets: one exporting the wavefront to openPMD output and the other one containing SimEx code. The wavefront output, as shown in Fig. 9 (a), is generated in the format conforming the openPMD Domain-Specific Naming Conventions for Photon Raytracing Codes and then used as a input in the diffraction simulation within the SimEx framework. The diffraction simulation is conducted on a single crystal Cu sample using the GAPD diffraction calculator, the result is presented in Fig. 9 (b).

Figure 9.

(a) The X-ray spectrum obtained from OASYS. (b) The diffraction pattern simulated with the X-ray spectrum in (a) and single crystal Cu sample using the GAPD calculator of SimEx.

This example has demonstrated how different backends in VINYL work together to complete the start-to-end simulation workflow.