You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "nuno-faria (via GitHub)" <gi...@apache.org> on 2025/08/01 11:45:30 UTC
[I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
nuno-faria opened a new issue, #17011:
URL: https://github.com/apache/datafusion/issues/17011
### Describe the bug
The `string_agg` function does not respect the `ORDER BY` clause in the `49.0.0` release. It does work after e1a5cdf6db791ea50b4028d6eb11ffafd940b29b I believe (#16625).
I discovered this while attempting to upgrade `datafusion-python` to `datafusion 49.0.0`, which causes some tests to fail. Would it be possible to release a minor update to allow `datafusion-python` to support `datafusion 49`?
### To Reproduce
On the released `49.0.0` version:
```sql
> create table t (k varchar, v int);
0 row(s) fetched.
Elapsed 0.055 seconds.
> insert into t values ('a', 2), ('b', 3), ('c', 1);
+-------+
| count |
+-------+
| 3 |
+-------+
1 row(s) fetched.
Elapsed 0.065 seconds.
> select string_agg(k, ',' order by v) from t;
+---------------------------------------------------------+
| string_agg(t.k,Utf8(",")) ORDER BY [t.v ASC NULLS LAST] |
+---------------------------------------------------------+
| a,b,c |
+---------------------------------------------------------+
1 row(s) fetched.
```
### Expected behavior
Should return:
```
+---------------------------------------------------------+
| string_agg(t.k,Utf8(",")) ORDER BY [t.v ASC NULLS LAST] |
+---------------------------------------------------------+
| c,a,b |
+---------------------------------------------------------+
```
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3189090373
We just verified it gets fixed in #17129 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3150036546
> It does work after [e1a5cdf](https://github.com/apache/datafusion/commit/e1a5cdf6db791ea50b4028d6eb11ffafd940b29b) I believe ([#16625](https://github.com/apache/datafusion/pull/16625)).
Thanks for linking!
That PR focused on `array_agg` and AFAIR added tests only for that funciton.
Would you be willing to contribute tests for `string_agg` as well, to make sure it does not regress in future releases?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151114351
Thanks for the feedback. Since the `string_agg` works with version `48.0.0` and below, I believe it could also fall under the "fix regressions" category.
My main need for the patch would be to upgrade `datafusion-python` to version 49, otherwise it must wait until version 50 (or remove the failing tests).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151945598
> Thanks for the feedback. Since the `string_agg` + `ORDER BY` works with version `48.0.0` and below, I believe it could also fall under the "fix regressions" category.
i didn't know it did. Then it's a correctness regression warranting a backport release.
I confirmed the following SQL returns correct results in 48.0.0 and incorrect result in 49.0.0 (lack of sorting)
```sql
create table t (k varchar, v int);
insert into t values ('a', 2), ('b', 3), ('c', 1);
select string_agg(k, ',' order by v) from t;
```
i got biased by the "It does work after https://github.com/apache/datafusion/commit/e1a5cdf6db791ea50b4028d6eb11ffafd940b29b" mention, which was supposed to be a new feature work, not a regression fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3150857828
@findepi Good idea, I have submitted a PR that adds logic tests for `string_agg` with `ORDER BY`.
About the bug in `49.0.0`, is a `49.0.1` release with the fix feasible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3189112531
> We just verified it gets fixed in [#17129](https://github.com/apache/datafusion/pull/17129) 👍
Note we (well really @AdamGS ) also backported the fix to 49.0.1: https://github.com/apache/datafusion/pull/17143
So it should be fixed in that release as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151007487
Thanks, approved!
Technically speaking this issue is a feature missing in 49.0.0 that already present on the main branch.
It will be released in 50.0.0, you can track it here: https://github.com/apache/datafusion/issues/16799
Releasing 49.0.1 is a good idea if we have bug fixes, especially for regressions found out after the release.
I feel your frustration -- the feature already exists, but it may be still a month or two until it is released and usable downstream. For that reason I am in favor of more frequent (and more automated!) releases, even every week. You can see prior discussion and chime in
- https://github.com/apache/datafusion/issues/5023#issuecomment-1399534610
- https://github.com/apache/datafusion/issues/14428
- https://github.com/apache/datafusion/issues/13661 (which is at odds with more frequent releases)
- https://github.com/apache/datafusion/issues/16622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3154185700
No problem, I should have rephrased that better.
Meanwhile, it appears that #16625 adds breaking changes, so it might not be simple to just cherry pick it to `49.0.1`. We might need to patch a new fix directly. I think the bug has been present since #16217.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3180232934
> `planning: Plan: ... does not satisfy order requirements ... `Just wanted to check if this could be related - in our case, the query doesn’t just return incorrect results, but it fails with the error. I can share a reproducer later & open a different issue if that’s helpful 🙇♀️ .
this sounds like a different issue -- a new ticket / reproducer would be most helpful. Thank you @LiaCastaneda
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151781169
> Thanks for the feedback. Since the string_agg + ORDER BY works with version 48.0.0 and below, I believe it could also fall under the "fix regressions" category.
Indeed, if this feature used to work and now does not I think it is a regression and we should fix it as a patch release
There were a few other items that people have suggested for inclusion. Here is a ticket to track a possible `49.0.1` release:
- https://github.com/apache/datafusion/issues/17036
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3180122308
👋 We’re seeing what might be a related error after upgrading to DF v49: `SanityCheckPlan caused by Error during planning: Plan: ... does not satisfy order requirements ... `Just wanted to check if this could be related - in our case, the query doesn’t just return incorrect results, but it fails with the error. I can share a reproducer later & open a different issue if that’s helpful 🙇♀️ . I tried using @nuno-faria’s PR to see if it fixed the issue, but it seems to fail in the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
LiaCastaneda commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3189507953
> > We just verified it gets fixed in [#17129](https://github.com/apache/datafusion/pull/17129) 👍
>
> Note we (well really [@AdamGS](https://github.com/AdamGS) ) also backported the fix to 49.0.1: [#17143](https://github.com/apache/datafusion/pull/17143)
>
> So it should be fixed in that release as well
Thank you 🙇♀️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3196092885
- Fixed in https://github.com/apache/datafusion/pull/17143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #17011: `string_agg` does not respect `ORDER BY` on `49.0.0`
URL: https://github.com/apache/datafusion/issues/17011
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151783347
Sounds like we should have tried harder to test datafusion-python with the 49.0.0 release candidate -- I think it just fell through the cracks
Thank you for helping drive this @nuno-faria
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3160722418
I backported the fix here: #17058.
I don't think the entire PR #16625 can be backported due to breaking changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `string_agg` does not respect `ORDER BY` on `49.0.0` [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #17011:
URL: https://github.com/apache/datafusion/issues/17011#issuecomment-3151016403
BTW recently there was a x.0.1 release:
- https://github.com/apache/datafusion/issues/16486
it looks it was in "fix release regressions" category
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org