You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "nuno-faria (via GitHub)" <gi...@apache.org> on 2025/08/01 11:45:30 UTC

[I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

nuno-faria opened a new issue, #17011:
URL: https://github.com/apache/datafusion/issues/17011

   ### Describe the bug
   
   The `string_agg` function does not respect the `ORDER BY` clause in the `49.0.0` release. It does work after e1a5cdf6db791ea50b4028d6eb11ffafd940b29b I believe (#16625).
   
   I discovered this while attempting to upgrade `datafusion-python` to `datafusion 49.0.0`, which causes some tests to fail. Would it be possible to release a minor update to allow `datafusion-python` to support `datafusion 49`?
   
   ### To Reproduce
   
   On the released `49.0.0` version:
   ```sql
   > create table t (k varchar, v int);
   0 row(s) fetched.
   Elapsed 0.055 seconds.
   
   > insert into t values ('a', 2), ('b', 3), ('c', 1);
   +-------+
   | count |
   +-------+
   | 3     |
   +-------+
   1 row(s) fetched.
   Elapsed 0.065 seconds.
   
   > select string_agg(k, ',' order by v) from t;
   +---------------------------------------------------------+
   | string_agg(t.k,Utf8(",")) ORDER BY [t.v ASC NULLS LAST] |
   +---------------------------------------------------------+
   | a,b,c                                                   |
   +---------------------------------------------------------+
   1 row(s) fetched.
   ```
   
   ### Expected behavior
   
   Should return:
   
   ```
   +---------------------------------------------------------+
   | string_agg(t.k,Utf8(",")) ORDER BY [t.v ASC NULLS LAST] |
   +---------------------------------------------------------+
   | c,a,b                                                   |
   +---------------------------------------------------------+
   ```
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3189090373

   We just verified it gets fixed in #17129 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3150036546

   > It does work after [e1a5cdf](https://github.com/apache/datafusion/commit/e1a5cdf6db791ea50b4028d6eb11ffafd940b29b) I believe ([#16625](https://github.com/apache/datafusion/pull/16625)).
   
   Thanks for linking!
   
   That PR focused on `array_agg` and AFAIR added tests only for that funciton.
   Would you be willing to contribute tests for `string_agg` as well, to make sure it does not regress in future releases?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151114351

   Thanks for the feedback. Since the `string_agg` works with version `48.0.0` and below, I believe it could also fall under the "fix regressions" category.
   
   My main need for the patch would be to upgrade `datafusion-python` to version 49, otherwise it must wait until version 50 (or remove the failing tests).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151945598

   > Thanks for the feedback. Since the `string_agg` + `ORDER BY` works with version `48.0.0` and below, I believe it could also fall under the "fix regressions" category.
   
   i didn't know it did. Then it's a correctness regression warranting a backport release.
   
   I confirmed the following SQL returns correct results in 48.0.0 and incorrect result in 49.0.0 (lack of sorting)
   ```sql
   create table t (k varchar, v int);
   insert into t values ('a', 2), ('b', 3), ('c', 1);
   select string_agg(k, ',' order by v) from t;
   ```
   
   i got biased by the "It does work after https://github.com/apache/datafusion/commit/e1a5cdf6db791ea50b4028d6eb11ffafd940b29b" mention, which was supposed to be a new feature work, not a regression fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3150857828

   @findepi Good idea, I have submitted a PR that adds logic tests for `string_agg` with `ORDER BY`.
   About the bug in `49.0.0`, is a `49.0.1` release with the fix feasible?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3189112531

   > We just verified it gets fixed in [#17129](https://github.com/apache/datafusion/pull/17129) 👍
   
   Note we (well really @AdamGS ) also backported the fix to 49.0.1: https://github.com/apache/datafusion/pull/17143
   
   So it should be fixed in that release as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151007487

   Thanks, approved!
   
   Technically speaking this issue is a feature missing  in 49.0.0 that already present on the main branch.
   It will be released in 50.0.0, you can track it here: https://github.com/apache/datafusion/issues/16799
   Releasing 49.0.1 is a good idea if we have bug fixes, especially for regressions found out after the release.
   
   I feel your frustration -- the feature already exists, but it may be still a month or two until it is released and usable downstream. For that reason I am in favor of more frequent (and more automated!) releases, even every week.  You can see prior discussion and chime in 
   - https://github.com/apache/datafusion/issues/5023#issuecomment-1399534610
   - https://github.com/apache/datafusion/issues/14428
   - https://github.com/apache/datafusion/issues/13661 (which is at odds with more frequent releases)
   - https://github.com/apache/datafusion/issues/16622
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3154185700

   No problem, I should have rephrased that better.
   Meanwhile, it appears that #16625 adds breaking changes, so it might not be simple to just cherry pick it to `49.0.1`. We might need to patch a new fix directly. I think the bug has been present since #16217.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3180232934

   > `planning: Plan: ... does not satisfy order requirements ... `Just wanted to check if this could be related - in our case, the query doesn’t just return incorrect results, but it fails with the error. I can share a reproducer later & open a different issue if that’s helpful 🙇‍♀️ .
   
   this sounds like a different issue -- a new ticket / reproducer would be most helpful. Thank you @LiaCastaneda 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151781169

   > Thanks for the feedback. Since the string_agg + ORDER BY works with version 48.0.0 and below, I believe it could also fall under the "fix regressions" category.
   
   Indeed, if this feature used to work and now does not I think it is a regression and we should fix it as a patch release
   
   There were a few other items that people have suggested for inclusion. Here is a ticket to track a possible `49.0.1` release:
   - https://github.com/apache/datafusion/issues/17036


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3180122308

   👋 We’re seeing what might be a related error after upgrading to DF v49: `SanityCheckPlan caused by Error during planning: Plan: ... does not satisfy order requirements ... `Just wanted to check if this could be related - in our case, the query doesn’t just return incorrect results, but it fails with the error. I can share a reproducer later & open a different issue if that’s helpful 🙇‍♀️ . I tried using @nuno-faria’s PR to see if it fixed the issue, but it seems to fail in the same way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3189507953

   
   > > We just verified it gets fixed in [#17129](https://github.com/apache/datafusion/pull/17129) 👍
   > 
   > Note we (well really [@AdamGS](https://github.com/AdamGS) ) also backported the fix to 49.0.1: [#17143](https://github.com/apache/datafusion/pull/17143)
   > 
   > So it should be fixed in that release as well
   
   Thank you 🙇‍♀️ 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3196092885

   - Fixed in https://github.com/apache/datafusion/pull/17143


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #17011: `string_agg` does not respect `ORDER BY` on `49.0.0`
URL: https://github.com/apache/datafusion/issues/17011


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151783347

   Sounds like we should have tried harder to test datafusion-python with the 49.0.0 release candidate -- I think it just fell through the cracks 
   
   Thank you for helping drive this @nuno-faria 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3160722418

   I backported the fix here: #17058.
   I don't think the entire PR #16625 can be backported due to breaking changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]

Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151016403

   BTW recently there was a x.0.1 release: 
   - https://github.com/apache/datafusion/issues/16486
   
   it looks it was in "fix release regressions" category


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org