Use Case #142
Enabling gravitational waves pipelines to process a stream of data from the ESCAPE datalake
The first use case we have identified is within the frame of enabling batch workflows for our gravitational wave (GW) pipelines.
For this purpose:
- We are proceeding with the containerization of data analysis pipelines and enabling them to process a stream of data from the ESCAPE datalake.
- To allow the easy deployment of VIRGO software inside containers, they needed to be made available via the Conda Package Manager, which required first that the software was buildable by CMake or Meson.
- Using the software that is deployed in Conda, we are able to create a container that downloads aggregated data from the datalake, process the data into 1-second files and stream the data to the GW pipelines.
For running these containers the deployment choice is Kubernetes from which a pilot project is ongoing at CNAF. This provides a realistic test of the GW pipelines to enable the offline analysis of data when deployed into any Kubernetes cluster available from computing centres.