1.

Introduction

The Square Kilometre Array Observatory (SKAO) is an international effort to build the largest and most sensitive radio telescopes. It is designed as a “physics machine” for the 21st century that will address fundamental scientific questions in astrophysics, fundamental physics, and astrobiology (see further details in the SKA science book¹). The SKAO will be a single observatory organized at three different sites, with the headquarters being located in the United Kingdom and two telescope sites, in Africa and Australia. SKA will be composed of thousands of antennas distributed over distances of up to 3000 km and thus constitute the largest radio telescope on Earth. Australia will host 131,000 low-frequency dipoles (SKA-low), operating between 50 and 350 MHz and reaching maximum baselines of up to 65 km; South Africa will host 197 dishes (SKA-mid), covering the frequency range from 350 MHz to 15.4 GHz (with the goal of reaching 24 GHz), with baselines up to 150 km.²

The construction phase will take place from 2021 to 2029, with science commissioning starting around 2023. SKA is recognized as one of the “big data” challenges for the next decade. By the time the SKA is in steady-state operations, SKA-low and SKA-mid will produce, respectively, a raw data rate of $\sim 2\mathrm{Pb}/\mathrm{s}$ (reduced to $\sim 7\mathrm{Tb}/\mathrm{s}$ before arrival to the central signal processor) and $20\mathrm{Tb}/\mathrm{s}$. The SKA science data processor (SDP) will deliver up to 300 PB of calibrated data (called observatory data products, ODPs) from each telescope to a network of SKA regional centers (SRCs).³ This model—a network of distributed facilities—has been used in the past by astronomy and particle physics communities to facilitate scientific analysis for state-of-the-art scientific instruments and infrastructures. The Large Hadron Collider (LHC) relies on a distributed computing infrastructure since its early days. Similarly, the Atacama Large Millimeter Array (ALMA) created a network of ALMA Regional Centers (ARCs) as an interface between the user community and the observatory. These centers are a key instrument to support scientific communities that carry out a variety of tasks, including preparation of observations, quality assurance of the data, archive operations, data reduction support, science operations, helpdesk, and commissioning.⁴

In the case of the SKAO, a combination of three main factors led to the decision of a distributed model: (1) the high data rates of the SKA telescopes, which impose extremely demanding computational requirements; (2) the data volumes, which are so large that direct delivery to users is not feasible; and (3) a globally distributed community that needs to work collaboratively.³ The SRC network will provide direct access to ODPs for an international community of collaborating teams and organizations, as well as to the tools and processing power for their transformation into science-ready data (i.e., advanced data products, ADPs) and subsequent scientific exploitation. Preliminary studies analyzed different computing infrastructure to optimize their use,⁵ and how they fulfill data processing requirements for the Low Frequency Array (LOFAR), one of the SKA-low pathfinders.⁶

The SRC network was conceived in September 2016, when the SKA organization established the SRC Coordination Group, with the mandate to define the requirements of the SRC network. This group (which includes LVM) identified an initial set of requirements associated with the six following areas: governance, science archive, accessibility and software tools, storage capacity, data processing capacity, and network connectivity.⁷ Furthermore, international projects and initiatives started contributing to the design of the SRCs, analyzing the requirements from a regional point of view. The European project “Advanced European Network of E-infrastructures for Astronomy with the SKA” (AENEAS),⁸ in which several of these authors participated, carried out, among other things, an estimation of the storage, computing, and data ingest capabilities required by a European SKA Data Centre. As well, they issued different reports with recommendations on the design of user interfaces for data discovery,⁹ data processing,¹⁰ and data transport.¹¹

Other examples of SRC initiatives are the “Exascale Research Infrastructure for Data in Asia-Pacific astroNomy using the SKA” project (ERIDANUS),¹² and the Canadian Initiative for Radio Astronomy Data Analysis (CIRADA).¹³ ERIDANUS is a design study focused on the prototyping of a data-intensive research infrastructure between China and Australia, and CIRADA is aimed at building the computing capacity required to scientifically exploit three radio telescopes [Very Large Array (VLA), Australian Square Kilometre Array Pathfinder (ASKAP), and Canadian Hydrogen Intensity mapping Experiment (CHIME)] and paving the way to a future Canadian SRC.

At the end of 2018, the SRC Steering Committee (SRCSC) took over from the SRC Coordination Group to take the SRC network from design to implementation on a global basis. This committee consists of one representative from each SKA member country that is carrying out activities to develop an SRC, as well as several representatives from the SKAO. At the time of writing, the countries participating in the steering committee are: Australia, Canada, China, France, Germany, India, Italy, Portugal, South Africa, Spain (represented by LVM), Sweden, Switzerland, The Netherlands, and United Kingdom. In addition, Japan and South Korea are observer countries in the committee. It is noteworthy that while the pace and the phase of the individual SRC developments in each country are different, in 2019, the Chinese SRC prototype was the first one to deploy dedicated infrastructure that was used to simulate SKA data transfer and processing.¹⁴ The steering committee is coordinating these national initiatives to develop the first global network of SRC prototypes. To this end, seven working groups were established, focused on the following areas: SRC network architecture, data logistics, operations, federated computing and data software services, SKA science archive, compute, and science user engagement.

Currently, some SKA precursor and pathfinder telescopes are already producing such large data volumes that distributing them to individual users is unfeasible. Instead, users are required to access a common service that provides the necessary framework. The archives and science platforms of these pathfinders are thus valuable testbeds for the SRCs. The scientific exploitation of the data generated by these telescopes requires developing computing centers where data processing can be centralized, minimizing the transfer of large volumes of data around the world, and with sufficient capacities to store them and to provide archive services. Examples include Ilifu in South Africa,¹⁵ which centralizes the processing of MeerKAT data in a cloud platform, and the Pawsey Supercomputing Centre in Australia,¹⁶ which provides specialized services for the ASKAP. The Canadian Advanced Network for Astronomy Research has built a science platform¹⁷ for data intensive astronomy that brings software to the data and provides user storage, user group management, or interactive and persistent virtual machines (VM), among other services. In Australia, the Commonwealth Scientific and Industrial Research Organization has built the ASKAP science data archive that provides access to data (i.e., images, image cubes, and catalogues) using both a virtual observatory and web services.¹⁸

The next few years are crucial to building expertise on processing and analyzing large and complex radio data to benefit scientifically from the SKAO telescopes. The community is working on it in different ways, e.g., via exploring synergies or participating in SKA science working groups. It is worth highlighting that the use of SKA precursors and pathfinder telescopes¹ is key for the community to get prepared for exploiting SKA and to optimize the scientific analysis of the SKAO data. In addition, a series of science data challenges are being released to the community by the SKAO¹⁹ since November 2018, and this is planned to continue until the end of 2024. The challenges entail analysis of simulated SKA data products that resemble the type of data that the SKAO will produce, with each separate challenge exploring a particular aspect of the SKA data products. The second data challenge receives support from international processing facilities where each participant team needs to use one computational facility to access and process the data mimicking how the future SRCs will work with the SKA data.^20,21 The second data challenge also includes a reproducibility award for those who demonstrate reusable methods and reproducible results.

SKA will be a world-leading facility and as such it aims to follow and lead best practices in scientific integrity as those promoted by the open science movement. The concept of open science facilitates the reproducibility of scientific studies by enhancing the accessibility, understandability, and reusability of their data and methods. Furthermore, open science has impacts on areas related to some of the United Nations Sustainable Development Goals. In particular, the UNESCO recommendation on open science acknowledges how this initiative contributes to democratize information, by fostering enhanced sharing of scientific knowledge among scientific communities while promoting inclusion of underrepresented or excluded groups. Also, it contributes to reducing inequalities in the access to infrastructures.²²

The adoption of open science values is rooted in SKA’s foundational principles,² with the aim to promote collaborative, transparent, and accessible scientific research. SKA is the first facility including reproducibility as a scientific metric of success, and the SRC network will preserve ODPs, ADPs, and the workflows and tools used to generate them. We have contributed to the inclusion of open science in key documents for the SKAO (e.g., SKAO prospectus or the SKA construction proposal²) and to including reproducibility as a metric that will be monitored during SKAO operations.²³ The SKAO and the SRC network are working to enable best practices that make data and other digital research objects (e.g., algorithms, tools, workflows, protocols, or services) “findable, accessible, interoperable, and reusable” (FAIR). In particular, the SRC Coordination Group defined different requirements related to open science,²⁴ highlighting the requirement of “open access,” which relates to the need for public links to SKA science data products, and the “reproducibility: provenance and workflow preservation” requirement, meaning that the SRCs must be capable of saving the provenance and the workflow associated with the data products generated at each SRC (see Sec. 5). Accordingly, open science introduces requirements into several SRCSC elements, and in particular to the SKA science archive. For example, the SKA Science Archive Working Group of the SRCSC is defining how SRCs will implement the FAIR principles and interoperate with the International Virtual Observatory Alliance (IVOA) ecosystem. The SKA-link project was led by the Instituto de Astrofísica de Andalucía (IAA-CSIC) and gathered key members of the SKA SDP consortium and SRC initiatives, together with experts on cutting-edge e-Science. The goal of the SKA-link was to identify best practices and technologies to successfully exploit SKA data in SRCs, with an emphasis on tools that facilitate the reproducibility. Furthermore, the “European Science Cluster of Astronomy and Particle physics ESFRI research infrastructures” H2020 project (ESCAPE)²⁵ brings together the SKA community with others in the astronomy and particle physics fields, aiming to build a multidisciplinary open environment in which they can share data, tools, and methods according to the FAIR principles. We participate in providing different use cases for the development of the ESCAPE Science Analysis Platform²⁶ and prototyping the integration of its resources into the European Open Science Cloud.

In this paper, we present the development of the Spanish Prototype of an SRC (SPSRC). The IAA-CSIC is the Spanish institution that coordinates the scientific and technological activities related to SKA in Spain since 2011. It is in close contact with the science community, e.g., by supporting and facilitating the participation in the SKA science working groups or organizing SKA-related. The SPSRC initiative came as a natural consequence of these coordination activities, combined with several decades of experience in radio interferometry, e-Science, and open science applied to radio astronomy. The research areas of the IAA-CSIC astronomers overlap with many of the areas in which the SKA will be fundamental,²⁷ making the IAA-CSIC an ideal environment for building the SPSRC. The research topics range from star and planet formation, trans-neptunian objects and centaurs, galaxy evolution (including star formation or radio-jets), the Milky Way center, transient phenomena to cosmic explosions. The application of Very Long Baseline Interferometric (VLBI) techniques (which provide the high angular resolution SKA will achieve) is transversal to all these fields.

The IAA-CSIC participated in the design of the SDP until it passed the critical design review, contributing to the SKA preservation and delivery subsystems. Since then, IAA-CSIC has contributed to defining, designing, and implementing the SRC network and took part in several of the initiatives mentioned above (SRC Coordination Group, SRCSC, AENEAS and ESCAPE projects, etc.). Currently, the collaboration with other SRCs through the SRCSC and its working groups is key to ensuring interoperability and the success of the SRC network. Furthermore, the IAA-CSIC carries out research and development activities in other areas of relevance for the SKAO and SRC network, i.e., e-Science and open science. For example, we have actively participated in the IVOA, developing software for data analysis and contributing to standards for astronomy; we contributed to developing the Astronomy Edition of the Taverna workflow management system,^28,29 and we participated in building the research object concept,³⁰ a powerful tool for achieving open science.

The SPSRC aims to support preparatory scientific activities for SKA key science projects and SKA precursors/pathfinder science while promoting synergies and best practices on open science and FAIR principles. Open science at the SPSRC not only aims to enable scientific reproducibility but also to open up the facility to the whole astronomical community beyond that of radio astronomy alone, contributing to build a transversal facility that facilitates wavelength-agnostic analysis. In the next sections, we explain the deployment of the hardware (Sec. 2) and the cloud infrastructure providing computing and storage resources (Sec. 3). Sections 4 and 5 describe the SPSRC cloud services and the tools enabling collaborative analysis as well as open science practices, including the prototyping activities to develop the science archive. The user support services and training activities are presented in Sec. 6, and the operational and management aspects of the SPSRC are described in Sec. 7. Finally, in Sec. 8, we discuss the results achieved so far and describe the developments to be accomplished in the near future.

2.

Hardware Platform

The technical specifications required for the SRCs have been estimated by the SRC Coordination Group as well as by the AENEAS project.^24,31 These specifications include aspects related to processing and storage capacity or network connectivity. They are further detailed in the SRC global size estimations document³² for computing and storage. The AENEAS project conducted an analysis of computing load, data transfer, data storage, and network requirements for the European SKA Science Data Centre,^33–36 including comparisons between different scenarios and technologies.

The hardware characteristics of the SPSRC are based on the previously mentioned reports. In addition, we surveyed 19 researchers representing seven IAA-CSIC research groups as a representative sample of the use cases of the IAA-CSIC (see Sec. 1). They represent potential future SPSRC users, and we gathered their current and future hardware and software requirements for data processing and analysis and were informed on their computational methodologies. In the second stage, we plan to investigate what requirements are needed for a wider community at a national level. We also conducted face-to-face meetings with the seven research groups to discuss the details of their particular science cases and data processing needs. In particular, data from the following topics were collected during these meetings:

During the interviews, the degree of radio astronomy expertise was assessed to identify areas that would require science support from the SPSRC team. In addition, we identified which radio telescopes are commonly used by the groups to know the type of data analysis that these groups would conduct at the SPSRC.

Once we had gathered and analyzed the information from the survey and the interviews, we discussed potential implementations that would satisfy the hardware and software requirements of most of the groups. The emphasis was on those working with radio data and in particular with data from SKA pathfinders and precursors. We considered what options would be the least disruptive for their current methodologies to ensure a smooth transition. We also took long-term feasibility to expand and maintain the infrastructure into consideration. Our conclusions for the technical requirements are:

This analysis of the technical requirements helped us make an informed decision on how to distribute the resources between CPU, memory, storage, and networking within the available budget. It was also considered the preliminary requirements in the SRC White Paper,³⁷ which was in preparation then. In particular, we decided to prioritize a large, high-performance storage capacity and to set up an infrastructure with high horizontal scalability (scale-out) that will allow to easily increase its computing capacity in future expansions.

From the survey, none of the ongoing or imminent projects explicitly required GPUs, so we did not prioritize including GPUs in the infrastructure in this first stage. However, given the relevance of the new data processing techniques that require GPUs (e.g., machine learning and deep learning techniques) and how they can be used to address the big data challenge in areas such as object detection and classification, we plan to include GPU nodes in the next hardware expansion.

After the requirements analysis and design phase in late 2019, we completed the procurement and deployment of the hardware in 2020. This was placed in a rack as shown in Fig. 1. The figure shows the distribution of the equipment and the organization of the deployment of cloud computing components, which will be detailed in Sec. 3.

Fig. 1

Rack distribution (upright for space reasons).

3.

Cloud Computing Infrastructure

In previous works, we have tested different cloud infrastructures, concluding that they are able to fulfil the requirements for processing data from the SKA pathfinder LOFAR. As well, they are offering interesting functionalities such as a straightforward and simple installation and maintenance of the software and the flexibility to adapt the infrastructure to the needs of the problem, among others.⁶ These functionalities cover the needs of the potential SPSRC users as explained in Sec. 2. Thus, to govern all the SPSRC hardware, we deployed the most widely used enterprise/science platform for cloud computing, i.e., OpenStack.⁴¹ The following sections detail how OpenStack has been deployed and configured in the SPSRC.

3.1.

OpenStack Deployment

OpenStack is one of the most active open-source projects in the world, being one of the essential components for the creation of private and public Cloud infrastructure.⁴² OpenStack has become the de facto standard for implementing cloud computing platforms through which to deliver infrastructure as a service (IaaS) that will, in its turn, finally deliver the entire stack of services related to platform as a service and software as a service in an extremely flexible and reliable way. With OpenStack, pools of computing, storage, and network resources are governed, with virtually infinite scaling capabilities. This is done via a command interface, API, and a control panel, which facilitate all its management.

We deployed the OpenStack version Train⁴³ within the computing, network, and storage infrastructure of the SPSRC described in Sec. 2. An important consideration when deploying this IaaS is the possibility of future scaling of resources and especially of managing them in a comfortable way, given the number of components. Thus, we use Kolla and Kayobe⁴⁴ as building and deployment tools for OpenStack.

These tools allow the deployment of OpenStack in containers on top of on-premise machines, as shown in Fig. 2. The advantage of this type of deployment is clear⁴⁴: (a) containers (such as docker⁴⁵) offer an attractive solution to isolate and distribute all OpenStack services throughout our SPSRC infrastructure efficiently, allowing to have production and test environments on the same infrastructure managing different versions of services without interfering with each other, (b) they enable a fast provision of nodes from a single startup host, in this way, the entire orchestration of services and hosts is centralized, and (c) it is recommended to build systems designed around performance, sustainability, reliability and scalability, basic principles of robust container deployment, and key features for the design pillars of this SPSRC. Kayobe uses Ansible⁴⁶ to perform system configuration and then invokes Kolla–Ansible to perform the control plane service deployment. The latter also oversees the management of every level of the system stack, as well as the hardware life-cycle. With the Ansible playbooks and environment files in source control, a comprehensive infrastructure-as-code⁴⁷ system is delivered for a fast provisioning. The source code for the SPSRC deployment is hosted and maintained in a private Gitlab repository.

Fig. 2

Kayobe deployment for OpenStack.

3.2.

OpenStack Architecture

To tackle the design of the OpenStack deployment architecture, three types of requirements have been taken into account: project constraints (computing and storage), operational requirements (networks, service level agreements, and maintenance), and high availability (data, services, and network). Based on these requirements, a logical architecture was designed in which the following core services have been deployed within OpenStack:

• (a) Computing:

  • • Nova: service that enables the provision of computing instances (virtual servers).

  • • Magnum: provides container orchestration engines for container deployment and management.

• (b) Storage:

  • • Cinder: block storage service to deliver volumes to VM deployed with Nova.

  • • Glance: component to upload and discover data assets such as operating system images.

  • • Manila: shared file system service to provide a file sharing platform as a service.

• (c) Network and authentication:

  • • Neutron: services and tools to provide network connectivity as a service between interface devices and the rest of the OpenStack services.

  • • Keystone: a component in charge of managing authentication and authorization of services and users.

• (d) Service level agreements and maintenance

  • • Monasca: provides a highly scalable, performance, and fault tolerant monitoring-as-a-service solution. It also offers an extensible platform for advanced monitoring of all OpenStack services.

  • • Horizon: enables a web interface for the control panel of all user-manageable services, such as nova, neutron, cinder, glance, and others.

In addition to these core services being deployed, other services necessary to support and monitoring the architecture have been used:

• • Core support services

  • – MariaDB Galera Cluster: a database for high availability.

  • – RabbitMQ: a message passing broker service for service intercommunication.

  • – Memcached: data caching for distributed memory.

  • – HAProxy: a tool for load balancing and high availability of services.

• • Monitoring components

  • – Prometheus: a monitoring and alerting toolkit.

  • – Grafana: a query, visualization, and alert GUI to summarize infrastructure data.

  • – Kibana: a search and data visualization tool for data indexed in Elasticsearch.

Figure 3 shows the set of OpenStack services deployed within the SPSRC infrastructure. Each of the blocks shown in the diagram are logical elements of OpenStack deployment. Each of these blocks (storage, monitoring, computing, and controllers) are a group of physical nodes with a set of replicated services (cinder, manila, nova, etc.), which supports high availability to the entire infrastructure, thus not having a single point of failure. The diagram also shows the two networks currently active: 1 GbE for infrastructure management and monitoring and 100 GbE for high speed communication between computing and storage. Figure 1 shows the layout of all these nodes inside the SPSRC rack.

Fig. 3

OpenStack architecture, logical design, nodes, network, and services.

4.

Cloud Services

4.1.

On-Demand Computing Resources

The SPSRC cloud computing infrastructure allows researchers to use different configurations of VMs on demand. Several critical points define each configuration, in particular:

• • Operating system: We have enabled a selection of the most popular Linux operating systems in the astronomy and astrophysics context. Via the OpenStack’s Glance service, we offer the following Linux distributions to be used within VMs: CentOS 7/8, Ubuntu 18.04/20.04, and Debian 9/10.

• • VM resources configuration: In OpenStack, flavors define the computing, memory, and storage capacity of the computing instances for the Nova service. A flavor is a hardware configuration available to a server that defines the size of a virtual server that can be launched. Figure 4 shows the set of configurations available to create VMs. As shown in the figure, three types of environments can be deployed, (a) general purpose VMs, with very flexible configuration possibilities ranging from light workload VMs to high load and heavy workload configurations, (b) VMs for SKA data challenge environments, with medium and high workloads, and (c) VMs for cluster compute nodes, such as Kubernetes or queue management systems such as Slurm. The flavor with which a VM is created is not a limitation because, if required, the flavor can be changed easily at any time. That is, the resources available to a VM can be scaled up or down depending on changing project necessities and the server load.

• • Mass storage: By default, all VMs have their root disk (i.e., the virtual hard drive where the operating system is installed) placed on the local SSD of a hypervisor (the VM monitor that runs the VM) to enable maximum performance. In addition to the default storage of each VM instance, it is possible to include two types of storage:

  • – Block storage (cinder.volumes): This type of storage is the most common for downloading and working with large data sets. It allows users to create volumes of any size, from megabytes to terabytes, that are mounted on VMs. This type of storage can be detached from one instance and reattached to another while the data remain intact, although it cannot be shared by more than one instance. It does, however, allow the use of extralarge volumes to store data and offers an excellent read/write performance.

  • – Shared file system storage (manila.ceph): This storage model is available for use in VMs that want to access a shared data or software repository or a service marketplace that can be shared by different users and services. The main advantage is that this type of storage can be shared by many VMs, which gives a great flexibility in working collaboratively. All VMs that are created can include a number of shared file system spaces for different purposes. In particular, we have deployed a shared file system to store a catalog of container images with software ready to use by any of the projects hosted in the SPSRC (Sec. 4.4).

• • Networking: All VMs have a public IP and their own “fully qualified domain name (FQDN)”, so they are directly accessible from the internet through a domain, following this pattern: vm_name.iaa.csic.es, for example, spsrc01.iaa.csic.es (Fig. 5). VM users belonging to the group spsrc-group can set up services within a predefined and strict range of ports, where only a subset of ports are open to outside.

• • Access mode to VM resources: Each VM can be accessed through two different protocols: SSH and RDP. By default, all VMs provide SSH access to users through public/private keys. In addition, Apache Guacamole is installed on the VMs. This tool provides a clientless remote desktop gateway. That is, it offers streamlined connection to VMs from any browser, supporting standard protocols, such as VNC, RDP, and SSH.

• • Core software and science services: Two sets of software are loaded onto each VM during creation. One is the software of the selected operating system, which includes the basic services, access/management accounts, monitoring services, and a desktop that are installed using Ansible. The other set consists of services for science such as JupyterHub and a container hub of containerized astrophysics and astronomy software as well as services that are installed on demand by the user. These last two service types are detailed in the following subsections.

Fig. 4

List of flavors and features within OpenStack.

Fig. 5

Network and external connectivity diagram.

Figure 6 shows the general process of creating a VM, including the stages previously detailed.

Fig. 6

The process of instantiating a VM in the SPSRC infrastructure.

5.

Science Archive

As stated in Sec. 1, the SKA will generate the largest amount of radio-astronomical data to date and the data products coming out of the observatory will be ingested into the SRC network at a rate of 600 PB/year. The SKA science archive is currently being designed as a distributed but logically centralized system that aims to provide fully functional and continuous capabilities.³⁷ The size and complexity of the SKAO data will necessarily impact the design of the archive and its connection to other subsystems that enable science analysis. The archive shall provide enough capabilities to search and explore data directly, to minimize data movement through the SRC network. It is primordial fulfilling the FAIR⁵⁶ principles and being compliant with the virtual observatory recommendations. The SKA science archive working group of the SRCSC is responsible for the definition of the archive requirements as well as for the design and prototyping of its main components. At the time of writing, this group is led by the Spanish and Canadian members of the SRCSC, and it consists of 35 people, which also includes a core group of 18 experts from Australia, France, Italy, The Netherlands, Portugal, SKAO, Spain, Sweden, and the United Kingdom.

In this section, we focus on the activities related to the SPSRC science archive, which comprises the storage space to preserve raw and processed data and the subsystems for searching and accessing the data. Considering that the SKA Observatory is starting the construction phase this year (2021), the SPSRC archive is initially being designed to satisfy the requirements of SKA precursors/pathfinder science cases, with full ability to scale up for SKA. The handling and sharing of the expected amount of data with the (authorized) scientific community demands careful planning on hardware characteristics and which services will be implemented to access archived resources.

As described in Sec. 2, we conducted a series of interviews to define the detailed requirements for this phase. With a capacity of 400 TB of net file storage dedicated to the archive, the hardware characteristics described in Table 1 will allow for the required $I/O$ rates to access and process the data from the SPSRC computational environment. High $I/O$ rate is one of the most important requirements, due to the large size of the data, which consequently impacts data processing times and the feasibility of deploying analysis services on top of the archive. The SPSRC archive will include advanced services on top of the infrastructure, such as user-archive interface services (supporting authentication, authorization, and querying), data and metadata visualization, cut-out services, a web portal, access from virtual observatory services, APIs (i.e., via scripts), etc. Some of these services require a significant amount of computational resources due to the nature of the data from the SKA precursors. The resilience of the subsystem is another key aspect that the SPSRC has addressed by including different levels of redundancy, i.e., at hardware and storage levels.

The subsystems to allow searching and accessing the data are an ongoing development. After testing existing virtual observatory libraries and software, we are currently deploying DaCHS in the infrastructure. DaCHS is an integrated framework for building web services connected to the Virtual Observatory ecosystem while supporting the entire workflow from ingestion to data mapping to service definition.⁵⁷ The SPSRC will host a series of services that will be registered at the European Virtual Observatory (EURO-VO) registry to ensure that they are discoverable from the astronomy software ecosystem. (IVOA Registries have information of all Virtual Observatory known web services. See Ref. 58.)

In particular, the SPSRC will provide access to catalogs using web services that enable SQL-like queries⁵⁹ to tabular data. These kinds of services can be queried from Virtual Observatory desktop software^60,61 and from libraries accessing Virtual Observatory services.^62,63 Images and data cubes will be findable from services compliant with the IVOA Simple Image Access recommendation.⁶⁴ In addition, we are developing specific services to provide access to derived radio data such as cut-outs (subsets of full data sets), by following the IVOA Server-side Operations for Data Access (SODA) recommendation,⁶⁵ a standard to develop services that perform operations on the data. Catalogs and image services will be implemented on the DaCHS framework whereas the SODA services will be developed separately. Interoperability with the rest of astronomy archives is fundamental to create a transversal facility that aims to enhance research and facilitate multimessenger science based on Open Science/FAIR practices.

As part of the SPSRC Science Archive developments, we are contributing to the development of the archive for the ASKAP WALLABY Hi survey.⁶⁶ This work is undertaken by its technical working group 7, and several copies of this archive are planned to be hosted at the SPSRC, Australian SRC, and in Canada. The WALLABY team has extended the capability of SoFiA (source finding software for three-dimensional spectral line data) by developing an automated, parallel HI source-finding pipeline for the WALLABY survey (SoFiA 2).⁶⁷ SoFiAX executes SoFiA-2 and automatically merges, resolves, and inserts extracted sources into a database for further inspection and analysis. The resulting cubelets, moment maps, or spectral data are accessible via a table access protocol (TAP) interface. At the time of writing, we are experimenting with a containerized DaCHS deployment.⁵⁷

Considering that SKA will not generally preserve raw data due to size issues, it is especially relevant to understand the processes and data transformations happening in the SKA SDP. With the aim to provide archive services that contribute to a better understanding of the data, we have studied how provenance data models can be applied to the SKA science data products and to the derived products that will be created at the SRCs. The term provenance refers to capturing the whole process involved to obtain the ADPs, such as actions, pipelines, versions, and parameters. This action of preserving the workflows used to generate the SKA data products and their provenance is an important step toward SKA science reproducibility. In collaboration with GMV Aerospace and Defence S.A.U, we have compared the ProvONE data model,⁶⁸ the W3C data model recommendation, and the IVOA data model recommendation.⁶⁹ ProvONE and the IVOA data models are both based on the W3C standard. The ProvONE extension was designed for a better description of scientific workflows and in particular its computational processes, whereas the IVOA recommendation is aiming to bring the provenance data model to the astronomy domain by adding new elements that are domain-specific. As a result of this study, we have concluded that the IVOA provenance data model can be applied to the SKA ⁷⁰ in a similar way to its application to a Cherenkov Telescope Array provenance implementation.⁷¹ However, the SKA pipelines and computational workflows are so complex that there may be a benefit using a combination of two models or a modification of the IVOA recommendation.⁷⁰

6.

Support and Training

Our aim is to support preparatory activities for the SKA key science projects and the science enabled by the SKA precursors and pathfinders. Therefore, the SPSRC has allocated human resources to support the projects hosted in the platform. Support and training are key to optimize the throughput of the platform by helping users to make efficient use of the available software and hardware resources, to gather new community requirements and to train the community on how to manage the data in the new data processing paradigm required by the SKAO. The SPSRC provides support and training on radio interferometry, state-of-the-art software tools, and automation of data processing workflows, all to enhance the scientific throughput of the platform.

The user support and helpdesk service are delivered through different channels to ensure flexible and effective communication. First, we have developed documentation providing information for users and a detailed description of the most useful procedures.⁷² Second, we use a dedicated SPSRC email address to deliver announcements, as well as to receive and solve questions from the users. This is complemented with a Slack workspace for the SPSRC in which we give technical and scientific support, including specific support in private channels for each project we host. When required, we also organize videocons and face-to-face meetings to discuss particular issues that users want to clarify. Last, we created a repository of singularity images with radio astronomical software to be used from the different VMs.

The most common support questions are on technical issues related to configuration and usage of the platform. We also help users to get familiar with and use the services provided, for example, how to use Guacamole as an access point, how to use JupyterHub to execute notebooks interactively, and how to access our SPSRC repository of singularity images. From the science point of view, we help users to access data from different observatories, find resources describing data processing pipelines for different instruments, and clarify general questions on radio interferometry, calibration, or potential use of instruments. Examples of these are discussions on best practices to access and retrieve MeerKAT data or discussions on how the calibration procedures of the MeerKAT pipeline oxkat⁷³ work, among other topics. To motivate the use of open science tools, we also give support and provide examples on tools, such as git, Github, conda software management, Jupyter notebooks, Binder, and Zenodo.

Apart from direct support using the communication channels described above, we also organize training events to promote the use of these tools and good practices related to them for a wider community. For example, we have conducted a series of short hands-on training events called Open Science Droplets,⁷⁴ describing and showing the usage of open science tools. We are also currently preparing two training schools: one focused on open science and another one on radio interferometry. The goal of these programs is to motivate and facilitate the community to adopt more flexible, efficient, collaborative, open tools. Apart from being a useful asset for conducting their own research, these practices will enhance the quality of the future SKA science.

7.

SPSRC Operations and Management

The SPSRC infrastructure is actively managed by a team with different purposes. First, to host a series of development and scientific projects that make use of those resources and services. Second, to enable the community to access the platform and get the support and training needed. Third, to ensure the daily operations of the available services. Lastly, to upgrade the software resources and to test and deploy new functionalities and services. Here, we summarize the current usage of the platform by development, scientific, and training programs as well as describing how the SPSRC is managed and organized.

7.1.

Usage of the SPSRC Services

The ultimate purpose of setting up all the hardware, software, services, and support described above is to provide a platform to conduct research projects. The SPSRC started to host the first science projects at the end of 2020. Once the deployment of the OpenStack server was finalized, by September 2020, a commissioning period started, until the end of 2020, to benchmark the disk and CPU usage of the VMs, deploy access services, and to verify all the functionalities of those services.

During the lifetime of the platform thus far, we have hosted 20 projects: 16 requiring VMs and four requiring JupyterHub deployments. We classify the hosted projects in three categories (see Fig. 7): (1) research projects, led by the IAA-CSIC researchers to conduct data processing and analysis on different astrophysical fields; (2) development projects, which have the goal of creating or enhancing services needed in an SRC and improving the platform; (3) training projects that include data processing or analysis schools and also temporary machines to be used during other observatory schools or to verify the capability of the platform for specific purposes (see Sec. 6)

Fig. 7

Overall view of the hosted SPSRC projects.

The research projects currently hosted at the SPSRC include analysis of data from the SKA precursor MeerKAT: observations of radio emission induced by star-planet interaction. We also host an e-MERLIN (SKA pathfinder) legacy project on resolved radio emission of ultraluminous infrared galaxies,⁷⁵ and a project on star-planet interaction based on LOFAR (SKA pathfinder) observations. Other research projects include, for example, the analysis of galactic radial profiles.

As some examples of development projects, we have deployed an online file repository and are developing an archive platform compatible with the virtual observatory (see Sec. 5), and we host a project to test the installation and execution of data processing pipelines for SKA precursors and pathfinders, among other projects. As part of a master’s thesis, we conducted a study of the data transfer quality between different international research centers using network tests.

The training projects at the SPSRC cover a wide range of services. We have hosted a JupyterHub service for two editions of SOMACHINE, which is an international school on machine learning, big data, and deep learning in astronomy. Likewise, we deployed temporary VMs to allow users to participate and execute the tutorials of the sixth LOFAR Data School. In addition, we are one of eight facilities providing infrastructure for the second SKA Data Challenge (see Sec. 1) issued by the SKA Observatory, hosting two teams.

Currently about 80% of the CPUs and memory are assigned to active projects. The usage has been continuously increasing since the beginning of the year. In terms of storage, until the archive services will become available (see Sec. 5), the main use is processing storage via block storage, with a mean assignation of 12 TB per month, on average. Current projects have required transferring and storing 110 TB of raw data. We have received requests to transfer a volume of 338 TB of raw data in the near future. Subsequent data processing usually requires disk space between one and five times the raw data volume. Apart from the block storage assigned to individual projects, we have deployed a shared storage unit that hosts a central service of software containers, currently including a catalog of singularity images with radio astronomical software ready to be used by any user from the different VMs.

8.

Conclusions and Future Work

Preparatory activities for the SRCs started in 2016 with the SRC Coordination Group, and they increased with the AENEAS and ERIDANUS projects. In Spain, the development of a prototype is led by the IAA-CSIC. One of the first activities was to gather requirements from a variety of science cases (especially those related to SKA precursors and pathfinders). In 2019, we interviewed seven groups as a representative sample of use cases. Then, we designed the SPSRC computing infrastructure to satisfy the requirements identified in those interviews, together with the SRC Coordination Group and AENEAS requirements. As a result, the SPSRC infrastructure is flexible and can redistribute resources. In addition, it can be adapted to heterogeneous projects and will support the science with SKA precursors and pathfinders. These abilities are key to identify (and solve) potential bottlenecks in the future SKA data processing. Moreover, the scalability offered by this infrastructure will allow increasing the SPSRC resources steadily until fulfilling the computing requirements for the next phases of the project and those that a fully operational SKA will introduce by (around) the year 2029.

The set-up of the SPSRC computing infrastructure was finalized in July 2020, followed by a commissioning period that lasted until the end of that year. During 2021, we gave progressively access to different research groups to deploy their own software and at the same time services like we deployed JupyterHub on the infrastructure. In the lifetime of the platform, it has hosted 20 projects (including research, development, and training projects) that have required 16 VMs and four JupyterHub services.

The services offered by the SPSRC are oriented to facilitate collaboration with a globally distributed community and to enable the best possible scientific practices. The SPSRC team is making a special effort to develop a platform that enables scientists to follow open science practices. This effort is channeled through three main areas:

The establishment of a Spanish SRC will play an essential role in providing the scientific community with access to SKAO data, as well as processing capacity and tools to generate and analyze ADPs. This will be done in collaboration with other SRCs and as a consequence, relevant topics for future work and research can be derived from the collaboration to be established within the framework of the SRC network. This collaboration, in the coming years, will make it possible to interconnect and interoperate all the individual SRCs, offering a global distributed and fully resilient service. The SPSRC team will carry out open science and radio astronomy training activities to bring best practices to the community. As well, we will offer support to precursor and pathfinder projects as primary activities during the interim. A key improvement of the SPSRC infrastructure will be the inclusion of a data transfer node, which will enhance data transfers and connectivity to the SKAO and SRC network. Another key objective will be to complete the science archive development, with core services that will allow access to the data through virtual observatory protocols. Apart from these, other improvements will be considered, such as the provision of a marketplace service for the creation of VMs, the allocation of storage space, or the installation of software and services on demand, as well as the use of workflow management services such as Slurm.