You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Jefffrey (via GitHub)" <gi...@apache.org> on 2026/01/02 15:04:33 UTC

[I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Jefffrey opened a new issue, #19612:
URL: https://github.com/apache/datafusion/issues/19612

   ### Describe the bug
   
   Originally from this discussion: https://github.com/apache/datafusion/pull/19278#discussion_r2616514268
   
   It turns out aggregate functions can still be used with window frames even if they don't implement `retract_batch`, if the window frame is `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW`. And if the accumulator consumes the internal state as part of `evaluate`, it can return incorrect results.
   
   ### To Reproduce
   
   In `aggregate.slt`, try this:
   
   ```sql
   query ITRRR
   SELECT
       timestamp,
       tags,
       value,
       median(value) OVER (
           PARTITION BY tags
           ORDER BY timestamp
           ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
       ),
       percentile_cont(value, 0.5) OVER (
           PARTITION BY tags
           ORDER BY timestamp
           ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
       )
   FROM median_window_test
   ORDER BY tags, timestamp;
   ----
   1 tag1 10 10 10
   2 tag1 20 15 15
   3 tag1 30 20 20
   4 tag1 40 25 25
   5 tag1 50 30 30
   1 tag2 60 60 60
   2 tag2 70 65 65
   3 tag2 80 70 70
   4 tag2 90 75 75
   5 tag2 100 80 80
   ```
   
   We would expect those last two columns to be the same. However the actual results are:
   
   ```sql
   query ITRRR
   SELECT
       timestamp,
       tags,
       value,
       median(value) OVER (
           PARTITION BY tags
           ORDER BY timestamp
           ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
       ),
       percentile_cont(value, 0.5) OVER (
           PARTITION BY tags
           ORDER BY timestamp
           ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
       )
   FROM median_window_test
   ORDER BY tags, timestamp;
   ----
   1 tag1 10 10 10
   2 tag1 20 15 20
   3 tag1 30 20 30
   4 tag1 40 25 40
   5 tag1 50 30 50
   1 tag2 60 60 60
   2 tag2 70 65 70
   3 tag2 80 70 80
   4 tag2 90 75 90
   5 tag2 100 80 100
   ```
   
   - See last column; it exactly follows the input data because it keeps discarding its state
   
   ### Expected behavior
   
   I've identified the following aggregate functions which likely exhibit this behaviour:
   
   - percentile_cont
   - string_agg
   - distinct median
   
   median was fixed by https://github.com/apache/datafusion/pull/19278
   
   We need to fix these accumulators to not consume the internal state on evaluate. We also would want to update the documentation around accumulators to make this behaviour obvious so we don't fall into the same trap later.
   
   
   ### Additional context
   
   Comment on why consuming internal state is bad for windows: https://github.com/apache/datafusion/pull/19278#discussion_r2609476348


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Posted by "GaneshPatil7517 (via GitHub)" <gi...@apache.org>.
GaneshPatil7517 commented on issue #19612:
URL: https://github.com/apache/datafusion/issues/19612#issuecomment-3706839577

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #19612:
URL: https://github.com/apache/datafusion/issues/19612#issuecomment-3752922220

   We'll just need fix for distinct median to close this off


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Posted by "GaneshPatil7517 (via GitHub)" <gi...@apache.org>.
GaneshPatil7517 commented on issue #19612:
URL: https://github.com/apache/datafusion/issues/19612#issuecomment-3706081034

   @Jefffrey Can you assign this isssue to me i want to work on this...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Posted by "GaneshPatil7517 (via GitHub)" <gi...@apache.org>.
GaneshPatil7517 commented on issue #19612:
URL: https://github.com/apache/datafusion/issues/19612#issuecomment-3706969940

   Hey @Jefffrey Please Review this once again: https://github.com/apache/datafusion/pull/19618


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #19612:
URL: https://github.com/apache/datafusion/issues/19612#issuecomment-3706599028

   Feel free to comment `take` to assign this issue to yourself if you feel you can tackle this: https://datafusion.apache.org/contributor-guide/index.html#open-contribution-and-assigning-tickets


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed issue #19612: Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour
URL: https://github.com/apache/datafusion/issues/19612


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Accumulators which don't implement `retract_batch` can still exhibit buggy behaviour [datafusion]

Posted by "GaneshPatil7517 (via GitHub)" <gi...@apache.org>.
GaneshPatil7517 commented on issue #19612:
URL: https://github.com/apache/datafusion/issues/19612#issuecomment-3706898053

   hey @Jefffrey Please Review this : https://github.com/apache/datafusion/pull/19618


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org