[jira] [Updated] (AIRAVATA-3970) [GSoC] Adaptive Metadata Indexing and ATLAS Integration in Airavata Data Catalog

["Jayanth Vennamreddy (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRAVATA-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jayanth Vennamreddy updated AIRAVATA-3970:
------------------------------------------
    Description: 
Background:

Apache Airavata’s data catalog provides structured metadata storage but currently lacks workload-aware optimization strategies and efficient support for filter-heavy scientific metadata queries. 

With potential integration of external scientific datasets such as ATLAS (a molecular dynamics database containing ~1,900 protein simulations with rich structural, domain, and MD metrics metadata), the catalog must support more advanced retrieval patterns, including:

- Key-based lookups by PDB chain
- Multi-field metadata filtering (organism, domain classification, resolution, MD metrics)
- Batch metadata retrieval following similarity searches
- Scalable performance under increasing dataset size (10k–100k+ records)

Currently, metadata retrieval mechanisms are optimized for structured storage but do not incorporate workload-aware indexing or filter-efficient retrieval for domain-rich scientific data.

Problem:

Static indexing strategies do not scale effectively for scientific metadata workloads where query patterns vary across research users. Additionally, current APIs are optimized primarily for single-key access and do not support efficient bulk or filter-driven retrieval.

As Airavata evolves to support protein-scale metadata and similarity-search-driven workflows, improvements in schema design, indexing strategy, and retrieval efficiency become necessary.

Proposed Work:

1. Design and implement a normalized schema extension in Airavata’s data catalog to represent ATLAS-style protein metadata, including support for multi-value domain classification fields (ECOD, CATH, SCOP).

2. Implement a metadata ingestion pipeline to import ATLAS protein records (~1,900 entries) into the catalog and validate correctness.

3. Add query telemetry instrumentation to metadata APIs to capture:
   - Filter predicates used
   - Query latency
   - Result set size
   - Field access frequency

4. Based on observed workload patterns, implement an index optimization module that:
   - Identifies high-frequency filter fields
   - Recommends and creates appropriate secondary or composite indexes
   - Benchmarks performance before and after index creation

5. Implement a batch metadata retrieval API optimized for similarity-search-driven workflows, enabling efficient bulk fetch of protein metadata records.

6. Evaluate performance under synthetic scaling (10k–100k records) to measure query latency improvements and indexing overhead.

Expected Outcomes:

- ATLAS metadata fully represented in Airavata’s data catalog with proper schema support.
- Telemetry-driven index optimization workflow implemented.
- Demonstrated reduction in metadata query latency for filter-heavy queries.
- Support for bulk metadata retrieval following similarity searches.
- Benchmarks and documentation demonstrating scalability improvements.

This issue will serve as the tracking issue for the GSoC 2026 proposal and scope discussion.

  was:
Background:

Apache Airavata’s data catalog currently provides structured metadata storage but does not support workload-aware indexing or adaptive optimization strategies. With the integration of external scientific datasets such as ATLAS (molecular dynamics simulations of ~1,900 proteins with rich structural and domain metadata), the catalog must support advanced retrieval patterns including:

- Key-based lookups by PDB chain
- Multi-field metadata filtering (organism, domain classification, resolution, MD metrics)
- Batch metadata retrieval following similarity searches
- Potential cross-system integration with sequence/structure search tools

Problem:

Static indexing strategies do not scale well for scientific metadata workloads where query patterns vary significantly between research users. Additionally, current retrieval APIs are optimized for single-key access and do not support efficient bulk or filtered access.

Proposed Work:

1. Extend Airavata’s data catalog schema to support protein-scale metadata (including multi-value domain classification fields such as ECOD, CATH, SCOP).
2. Implement query telemetry to capture metadata access patterns.
3. Design and implement an adaptive indexing engine that:
   - Tracks frequently filtered fields
   - Dynamically creates or adjusts secondary indexes
   - Optimizes composite index creation based on workload statistics
4. Implement batch metadata retrieval APIs optimized for similarity-search-driven workflows.
5. Evaluate performance under synthetic scaling (10k–100k protein metadata records).

Expected Outcomes:

- Improved metadata retrieval latency
- Reduced scan overhead for filter-heavy queries
- Better support for scientific discovery workflows
- A foundation for scalable ATLAS-style dataset integration

This issue will serve as the tracking issue for the GSoC 2026 proposal.


> [GSoC] Adaptive Metadata Indexing and ATLAS Integration in Airavata Data Catalog
> --------------------------------------------------------------------------------
>
>                 Key: AIRAVATA-3970
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-3970
>             Project: Airavata
>          Issue Type: Improvement
>            Reporter: Jayanth Vennamreddy
>            Priority: Major
>
> Background:
> Apache Airavata’s data catalog provides structured metadata storage but currently lacks workload-aware optimization strategies and efficient support for filter-heavy scientific metadata queries. 
> With potential integration of external scientific datasets such as ATLAS (a molecular dynamics database containing ~1,900 protein simulations with rich structural, domain, and MD metrics metadata), the catalog must support more advanced retrieval patterns, including:
> - Key-based lookups by PDB chain
> - Multi-field metadata filtering (organism, domain classification, resolution, MD metrics)
> - Batch metadata retrieval following similarity searches
> - Scalable performance under increasing dataset size (10k–100k+ records)
> Currently, metadata retrieval mechanisms are optimized for structured storage but do not incorporate workload-aware indexing or filter-efficient retrieval for domain-rich scientific data.
> Problem:
> Static indexing strategies do not scale effectively for scientific metadata workloads where query patterns vary across research users. Additionally, current APIs are optimized primarily for single-key access and do not support efficient bulk or filter-driven retrieval.
> As Airavata evolves to support protein-scale metadata and similarity-search-driven workflows, improvements in schema design, indexing strategy, and retrieval efficiency become necessary.
> Proposed Work:
> 1. Design and implement a normalized schema extension in Airavata’s data catalog to represent ATLAS-style protein metadata, including support for multi-value domain classification fields (ECOD, CATH, SCOP).
> 2. Implement a metadata ingestion pipeline to import ATLAS protein records (~1,900 entries) into the catalog and validate correctness.
> 3. Add query telemetry instrumentation to metadata APIs to capture:
>    - Filter predicates used
>    - Query latency
>    - Result set size
>    - Field access frequency
> 4. Based on observed workload patterns, implement an index optimization module that:
>    - Identifies high-frequency filter fields
>    - Recommends and creates appropriate secondary or composite indexes
>    - Benchmarks performance before and after index creation
> 5. Implement a batch metadata retrieval API optimized for similarity-search-driven workflows, enabling efficient bulk fetch of protein metadata records.
> 6. Evaluate performance under synthetic scaling (10k–100k records) to measure query latency improvements and indexing overhead.
> Expected Outcomes:
> - ATLAS metadata fully represented in Airavata’s data catalog with proper schema support.
> - Telemetry-driven index optimization workflow implemented.
> - Demonstrated reduction in metadata query latency for filter-heavy queries.
> - Support for bulk metadata retrieval following similarity searches.
> - Benchmarks and documentation demonstrating scalability improvements.
> This issue will serve as the tracking issue for the GSoC 2026 proposal and scope discussion.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)