[jira] [Created] (GSOC-310) Apache SkyWalking: BanyanDB Native Data Export/Import Utility

["Hongtao Gao (Jira)" <ji...@apache.org>]

Hongtao Gao created GSOC-310:
--------------------------------

             Summary: Apache SkyWalking: BanyanDB Native Data Export/Import Utility
                 Key: GSOC-310
                 URL: https://issues.apache.org/jira/browse/GSOC-310
             Project: Comdev GSOC
          Issue Type: New Feature
            Reporter: Hongtao Gao


{panel}
h3. *Background*

*BanyanDB* is the native storage engine for Apache SkyWalking, designed specifically for observability data (Traces, Metrics, and Logs). As BanyanDB matures into a production-ready storage backend, data portability becomes critical. Users need the ability to move datasets between environments (e.g., from production to staging for debugging) or export data for external analysis in tools like Python/Pandas, Spark, or specialized AI training pipelines.

Currently, BanyanDB supports disaster recovery backups and simple CSV dumps for specific models. This project aims to build a high-performance, comprehensive *Export/Import Utility* that supports multiple formats and ensures data integrity.
h3. *Tasks*
 * *Multi-Format Support:* Implement export/import functionality for:

 ** *Native Binary:* High-performance format for BanyanDB-to-BanyanDB migration.

 ** *Plain Text/Standard:* Support for *Parquet* (optimized for metrics/measures) and *JSON/CSV* (for human readability).

 * *Batch & Stream Processing:* Ensure the tool can handle massive datasets by implementing chunked data reading and writing to avoid memory bottlenecks.

 * *Schema Evolution Handling:* Implement logic to handle cases where the schema in the exported file differs slightly from the target server's schema.

 * *Integration with {{{}bydbctl{}}}:* Expose these capabilities through a user-friendly CLI command suite (e.g., {{{}bydbctl data export --group=user_logs --format=parquet{}}}).

h3. *Requirements*
 * Strong knowledge of *Go* and concurrency patterns.

 * Experience with data serialization formats (Protobuf, Parquet, Apache Arrow).

 * Familiarity with gRPC-based API communication.

 
{panel}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: gsoc-unsubscribe@community.apache.org
For additional commands, e-mail: gsoc-help@community.apache.org