Project

General

Profile

Use Case #16

SKA Global Data Lake PoC for Astronomy-scale Data Products

Added by Rosie Bolton 3 months ago. Updated 21 days ago.

Status:
New
Priority:
High
Assignee:
Target version:
-
% Done:

0%


Description

Contact Person / Author: Rosie Bolton

Goal / Aim

Data management (SKA-own rucio), built on ESCAPE partner storage sites and using CERN FTS - important for SKA to test long-haul (inter-continental) links, to understand how rucio data management can fit SKA's use case for getting data into the SKA Regional Centres, and for managing replicas between data centres.


Subtasks

Product #17: Data Ingest into SKA rucioNew

Actions
Product #20: Repeat longhaul data transfer tests with globus as the transfer toolNew

Actions
Product #83: SKAO Data permissions use case feasibility assessmentNew

Actions
Product #87: Metadata functionality testsNew

Actions
Product #88: Metadata Performance testsNew

Actions
Product #89: SKAO Subscriptions testsNew

Actions
Product #90: End-to-end proof of SKAO data lifecycleNew

Actions
Product #91: Long haul transfers working with X509New

Actions
Product #18: Stress test the long haul links with small file size, capture throughputNew

Actions
Product #19: Stress test the long haul links with large file size, capture throughputNew

Actions
Product #92: (Uncommitted) Develop and test Containerised lightweight low-overhead RSENew

Actions

Add

Related issues

History

#1

Updated by Rosie Bolton 3 months ago

#2

Updated by Rosie Bolton 3 months ago

  • Copied from deleted (Use Case #13: Example: CTA PI Analysis)
#3

Updated by Rosie Bolton 3 months ago

  • Tags changed from Example:ESFRI to Example:ESFRI, SKAO, DAC
#4

Updated by Rosie Bolton 3 months ago

  • Tags changed from DAC, SKAO, Example:ESFRI to SKAO, Example:ESFRI, DAC21
#5

Updated by Jutta Schnabel about 2 months ago

  • Target version deleted (Example use case implementation)
  • Project changed from Using this plattform to ESFRI: SKA
#6

Updated by Rosie Bolton 21 days ago

  • Documentation changed from https://docs.google.com/document/d/1xofZ0-UdPbSA-tlM9X9q1Bm0h2K683YxNbTp0s40MOc/edit to https://docs.google.com/document/d/1xofZ0-UdPbSA-tlM9X9q1Bm0h2K683YxNbTp0s40MOc/edit#heading=h.wlvo5c1wugwt
#7

Updated by Rosie Bolton 21 days ago

  • Documentation changed from https://docs.google.com/document/d/1xofZ0-UdPbSA-tlM9X9q1Bm0h2K683YxNbTp0s40MOc/edit#heading=h.wlvo5c1wugwt to https://docs.google.com/document/d/1H255SitHzE8k4DLCWBmu8dosxAev0Vq9Sm0GzzK2Pg8/edit#
#8

Updated by Rosie Bolton 21 days ago

  • Description updated (diff)
#9

Updated by Rosie Bolton 21 days ago

  • Subject changed from Example: SKA Global Data Lake PoC for Astronomy-scale Data Products to SKA Global Data Lake PoC for Astronomy-scale Data Products

Also available in: Atom PDF