Project

General

Profile

Product #88

Use Case #16: SKA Global Data Lake PoC for Astronomy-scale Data Products

Metadata Performance tests

Added by Rosie Bolton 2 months ago. Updated 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Documentation:
Repository:
Tags:

Description

Motive
We want to know how performant the Rucio metadata is and know limitations

Considerations:
How many key/values to test? 100/1000? Measure time to retrieve a list of files based on x number of key/value pairs?

Theoretical exercise for SKA SRC Scale
Outcome - description of how JSON vs column metadata should be used, description of potential use of nested metadata to create "namespace" behaviour, description of scaling for astronomy metadata and SKA archive scale.

Notes: https://docs.google.com/document/d/1H255SitHzE8k4DLCWBmu8dosxAev0Vq9Sm0GzzK2Pg8/edit#heading=h.jnk7fih4f4e8


Add

Related issues

History

#1

Updated by Rosie Bolton 2 months ago

  • Tracker changed from Use Case to Product

Also available in: Atom PDF