[I] [Feature] Add Full Iterator Pattern to BanyanDB's Query Pipeline [skywalking]

["hanahmily (via GitHub)" <gi...@apache.org>]

hanahmily opened a new issue, #12913:
URL: https://github.com/apache/skywalking/issues/12913

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.
   
   
   ### Description
   
   
   
   BanyanDB's query pipeline currently utilizes the iterator pattern for sorting, aggregating, and limiting data. However, in the initial stage of the pipeline—the raw data retrieval—all data in the segments are loaded into memory. This approach can lead to excessive memory usage, especially for heavy aggregation queries, such as retrieving the top 10 items ordered by a tag over a large time range (e.g., "last month").
   
   We propose extending the iterator pattern to the initial raw data retrieval step to address this issue. By doing so, we can significantly reduce memory consumption by streaming data from segments on-demand rather than loading all segment data into memory at once.
   
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a pull request to implement this on your own?
   
   - [ ] Yes I am willing to submit a pull request on my own!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Add Full Iterator Pattern to BanyanDB's Query Pipeline [skywalking]

["hanahmily (via GitHub)" <gi...@apache.org>]

hanahmily closed issue #12913: [Feature] Add Full Iterator Pattern to BanyanDB's Query Pipeline
URL: https://github.com/apache/skywalking/issues/12913


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org