Project

General

Profile

Use Case #142

Enabling gravitational waves pipelines to process a stream of data from the ESCAPE datalake

Added by Rhys Poulton over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

60%

Documentation:

Description

The first use case we have identified is within the frame of enabling batch workflows for our gravitational wave (GW) pipelines.

For this purpose:

  • We are proceeding with the containerization of data analysis pipelines and enabling them to process a stream of data from the ESCAPE datalake.
  • To allow the easy deployment of VIRGO software inside containers, they needed to be made available via the Conda Package Manager, which required first that the software was buildable by CMake or Meson.
  • Using the software that is deployed in Conda, we are able to create a container that downloads aggregated data from the datalake, process the data into 1-second files and stream the data to the GW pipelines.

For running these containers the deployment choice is Kubernetes from which a pilot project is ongoing at CNAF. This provides a realistic test of the GW pipelines to enable the offline analysis of data when deployed into any Kubernetes cluster available from computing centres.

Files

ESCAPE_WP52_EGO_batch_use_case.jpg (74.2 KB) ESCAPE_WP52_EGO_batch_use_case.jpg Rhys Poulton, 11/30/2021 09:43 AM

Also available in: Atom PDF