You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "somu-imply (via GitHub)" <gi...@apache.org> on 2024/03/14 18:39:28 UTC

[I] Window functions on MSQ for high and low cardinality data (druid)

somu-imply opened a new issue, #16126:
URL: https://github.com/apache/druid/issues/16126

   At this moment, window functions in MSQ rely on materializing the data with the same partition into a rows and columns, process them and output to a frame. In case of high cardinality data (where a single partition has few distinct group and each group has an enormous amount of values) this can lead to running out of memory resources. One solution for this would be to redesign window functions in MSQ by not materializing the data and make multiple passes over the data. Most of the window operations like SUM(), RANK(), RANGE() etc. can be made with 2 passes of the data. We might think to reimplement them in the MSQ way so that we do not have to materialize so much data.
   
   Similarly for data with low cardinality (large number of distance groups, each with small number of rows), at this moment we add a single group to a frame. This would end up creating  a large number of frames. We can get past this by creating a fixed size frame and keep adding to the pending frame. The sorter will process the frames and make sure the data with the correct key reaches the correct processor


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Window functions on MSQ for high and low cardinality data (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16126:
URL: https://github.com/apache/druid/issues/16126#issuecomment-2561504333

   This issue has been marked as stale due to 280 days of inactivity.
   It will be closed in 4 weeks if no further activity occurs. If this issue is still
   relevant, please simply write any comment. Even if closed, you can still revive the
   issue at any time or discuss it on the dev@druid.apache.org list.
   Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Window functions on MSQ for high and low cardinality data (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #16126: Window functions on MSQ for high and low cardinality data
URL: https://github.com/apache/druid/issues/16126


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Window functions on MSQ for high and low cardinality data (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #16126:
URL: https://github.com/apache/druid/issues/16126#issuecomment-2608559466

   This issue has been closed due to lack of activity. If you think that
   is incorrect, or the issue requires additional review, you can revive the issue at
   any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [I] Window functions on MSQ for high and low cardinality data (druid)

Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on issue #16126:
URL: https://github.com/apache/druid/issues/16126#issuecomment-2006686677

   Base PR : https://github.com/apache/druid/pull/15470


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org