You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "hanahmily (via GitHub)" <gi...@apache.org> on 2024/11/29 22:45:10 UTC

[I] [Feature] Export Data Based on Backup Snapshots [skywalking]

hanahmily opened a new issue, #12815:
URL: https://github.com/apache/skywalking/issues/12815

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   
   #### Description
   
   This feature relies on the **Backup and Restore Data** functionality outlined in #12463. The backup functionality creates a consistent snapshot of the dataset. Building on that, the **Export Manager** will enable exporting data based on these snapshots, allowing users to extract and utilize specific sets of data efficiently.
   
   #### Requirements
   
   1. **Integration with Backup Snapshots**:
      - Leverage the snapshot created by the **Backup and Restore** feature as the data source for export.
      - Ensure the export process clean up the snapshot is created for the exportation.
   
   2. **Export Configuration**:
      - **Time Range Export**: Allow users to define a specific time range from the snapshot for export.
      - **Query-Based Export**: Allow users to filter and export data using specific query criteria (e.g., tags, metrics, identifiers).
   
   3. **Export Formats**:
      - Support commonly used data formats like **JSON** and **CSV**.
   
   
   
   
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a pull request to implement this on your own?
   
   - [ ] Yes I am willing to submit a pull request on my own!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup S3 Bucket [skywalking]

Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu closed issue #12815: [Feature] Export Data Based on Backup S3 Bucket
URL: https://github.com/apache/skywalking/issues/12815


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup Snapshots [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-2508824666

   About the feature, the exportation should support tags/fields selection if possible. This could reduce the volume costs of the files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup S3 Bucket [skywalking]

Posted by "prashanth9968 (via GitHub)" <gi...@apache.org>.
prashanth9968 commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-5096104131

   Hi @wu-sheng and @hanahmily! I am a student who is good at java with microservices architecture,backend development and dealing with strong formats.I see you are reserving this feature task for new developer programs like GSoC.I am very keen on participating and would love to start discussing the design details for this S3 exporting tool under your guidance.May i pick this up?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup Snapshots [skywalking]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-2508898445

   I want to create a design to explain the details. Perhaps we can set up exporting to the backup process.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup Snapshots [skywalking]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-2508755190

   > Is Export Manager going to be a separate role node?
   
   It is a command-line tool provided by `bydbctl`, which can be scheduled to run automatically using `cron` on a daily basis or at specified intervals.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup S3 Bucket [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-2644548196

   Exporting could be a separate tool, and it is not critical for the whole ecosystem. 
   This tool will read the whole snapshot from the s3 or efs, and filter out the data to generate exporting result.
   We will reserve this for new developer programs, such as GSOC or OSPP 2025


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup S3 Bucket [skywalking]

Posted by "hanahmily (via GitHub)" <gi...@apache.org>.
hanahmily commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-5725881705

   > Hi [@hanahmily](https://github.com/hanahmily) [@wu-sheng](https://github.com/wu-sheng), is this still held for GSoC or OSPP? The last word on that is from February 2025.
   > 
   
   > [@hanahmily](https://github.com/hanahmily), you mentioned wanting to write the design first. Is that still yours to write, or would a proposal from me be more useful to react to?
   > 
   > Happy to write one if it helps. If the issue is still spoken for, I will leave it be.
   
   No, it's not. @mrproliu is working on it. His design spec has passed the review phase. You can check out other issues. Thank you
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup Snapshots [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-2508756021

   So, the tool is going to grap snapshots from mutiple data nodes(through liaison)?
   Where the filter happens? Are they processed on data nodes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup S3 Bucket [skywalking]

Posted by "mrproliu (via GitHub)" <gi...@apache.org>.
mrproliu commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-5788487059

   I have [created a new one to track](https://github.com/apache/skywalking/issues/14102), so this issue has been closed. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup Snapshots [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-2508750180

   Is Export Manager going to be a separate role node?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup S3 Bucket [skywalking]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-5100840297

   > Hi @wu-sheng and @hanahmily! I am a student who is good at java with microservices architecture,backend development and dealing with strong formats.I see you are reserving this feature task for new developer programs like GSoC.I am very keen on participating and would love to start discussing the design details for this S3 exporting tool under your guidance.May i pick this up?
   
   This issue targets to the BanyanDB, which is a Go DB, not a Java.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature] Export Data Based on Backup S3 Bucket [skywalking]

Posted by "SammyUrfen (via GitHub)" <gi...@apache.org>.
SammyUrfen commented on issue #12815:
URL: https://github.com/apache/skywalking/issues/12815#issuecomment-5720541293

   
   Hi @hanahmily @wu-sheng, is this still held for GSoC or OSPP? The last word on that is from February 2025.
   
   @hanahmily, you mentioned wanting to write the design first. Is that still yours to write, or would a proposal from me be more useful to react to?
   
   Happy to write one if it helps. If the issue is still spoken for, I will leave it be.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org