You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/04/01 20:16:22 UTC
[I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
alamb opened a new issue, #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904
### Is your feature request related to a problem or challenge?
@universalmind303 had a request for inclusion in DataFusion 37.0.0 that may miss the release: https://github.com/apache/arrow-datafusion/issues/9900#issuecomment-2030482986
### Describe the solution you'd like
Make a `37.1.0` release with non breaking changes
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2059945930
> Heey @alamb, when are you planning this release? :)
@ion-elgreco I hope the end of this week -- I am going to do the backporting now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063656064
Here is a PR with the changelog / version update: https://github.com/apache/arrow-datafusion/pull/10128
```shell
./dev/release/generate-changelog.py apache/arrow-datafusion 37.0.0 branch-37 > dev/changelog/37.1.0.md
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2056728481
> That sounds good. My plan was to use branch-{version} for any maintenance releases, following the pattern used by Apache Spark., so we would release any 37.x.x releases from branch-37.
Makes sense -- I will do that.
I'll plan to make some backport PRs tomorrow morning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063879822
I see -- so from that perspective 37.0.0 and earlier have a bug where `int8/int16` bloom filters can incorrectly filter out incorrect answers.
I did some archeology and it seems like the in8/int16 bloom filter support was added in https://github.com/apache/arrow-datafusion/pull/7821 / shipped as part of https://github.com/apache/arrow-datafusion/blob/main/dev/changelog/33.0.0.md
(aka the bug has been present for several months/releases yet)
Thus I think it would be useful to fix but I also don't think it is critical to include in 37.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2065348963
Unfortunately it appears I made an error and forgot to update the release version in this RC
I have started a new thread[1] for a second RC https://lists.apache.org/thread/0md6qyhw0hody8p0v9wddvt7vo8r8z2x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063573607
We also added https://github.com/apache/arrow-datafusion/pull/10123 to this release, to reduce debug stack frame sizes. Thanks @sergiimk and @devinjdangelo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2059983116
I made the following 4 backport PRs: https://github.com/apache/arrow-datafusion/pull/10103, https://github.com/apache/arrow-datafusion/pull/10104, https://github.com/apache/arrow-datafusion/pull/10105, https://github.com/apache/arrow-datafusion/pull/10107
I will also send a note to the dev list and on slack/discord in case anyone else wants to backport additional items
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2059587021
Heey @alamb, when are you planning this release? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/datafusion/issues/9904#issuecomment-2069143242
The release was approved and published to crates.io: https://lists.apache.org/thread/v6y745zyljoor471964l3tfq8m37lzh2
The release is available here:
https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-37.1.0
It has also been released to crates.io: https://crates.io/crates/datafusion/37.1.0
Thank you to everyone 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #9904: Release DataFusion `37.1.0` (non breaking API release)
URL: https://github.com/apache/datafusion/issues/9904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063731029
> Would #9770 and #9969 be eligible for backport too?
@progval -- If it was a regression or a serious bug I think it would be a good idea to backport. In this case I can't remember the exact sequence of events (as in some types were enabled, and then disabled, but I am not sure what was in the 37.0.0 release)
What exactly do you think we should backport?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/datafusion/issues/9904#issuecomment-2069315395
> Thanks @alamb!
>
> I tried it with 37.1.0 in delta-rs, but we still get this error: ` internal error: entered unreachable code: NamedStructField should be rewritten in OperatorToFunction`, wasn't this regression fixed?
Hi @ion-elgreco -- another reason this could happen is that the array function's haven't been registered with your session context / however you are
Yeah you need to use the FunctionRewrtiter here (with the relevant rewriter registered) https://github.com/apache/arrow-datafusion/blob/0573f78c7e7a4d94c3204cee464b3860479e0afb/datafusion/optimizer/src/analyzer/function_rewrite.rs#L33
There was a similar question in discord here: https://discord.com/channels/885562378132000778/1166447479609376850/1229122082256851054
Can you point me at the code of how you are running your query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2059987844
Note that #9900 didn't in fact make all the udf's public - the string and unicode modules do not in fact have the modules changed to pub nor is default implemented for the functions that I spot checked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2052236820
> Thanks @alamb. I have more bandwidth now to help support maintenance releases. Do you have a target date in mind for 37.1.0?
Thank you !
I didn't have any date in mind -- how about we try for early next week (of April 15)? I am happy to do the backport PRs -- should we make a maintenance branch? Or maybe we should use https://github.com/apache/arrow-datafusion/tree/branch-37 🤔 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "progval (via GitHub)" <gi...@apache.org>.
progval commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063716977
Would https://github.com/apache/arrow-datafusion/pull/9770 and https://github.com/apache/arrow-datafusion/pull/9969 be eligible for backport too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on issue #9904:
URL: https://github.com/apache/datafusion/issues/9904#issuecomment-2069181423
Thanks @alamb!
I tried it with 37.1.0 in delta-rs, but we still get this error: ` internal error: entered unreachable code: NamedStructField should be rewritten in OperatorToFunction`, wasn't this regression fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]
Posted by "ion-elgreco (via GitHub)" <gi...@apache.org>.
ion-elgreco commented on issue #9904:
URL: https://github.com/apache/datafusion/issues/9904#issuecomment-2069359171
@alamb this is the code:
```
let (table, _metrics) = DeltaOps(table)
.delete()
.with_predicate("props['a'] = '2021-02-02'")
.await
.unwrap();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "progval (via GitHub)" <gi...@apache.org>.
progval commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063743396
int8/int16/int32/int64 were enabled for a while. #9770 enabled for uint8/uint16/uint32/uint64, then #9969 disabled for int8/int16/uint8/uint16.
So backporting #9770 adds a non-breaking feature and a regression; and backporting #9969 immediately fixes that regression plus an existing bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2049363574
I started collecting some potential fixes for this release (basically regressions) in the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2053809816
> > Thanks @alamb. I have more bandwidth now to help support maintenance releases. Do you have a target date in mind for 37.1.0?
>
> Thank you !
>
> I didn't have any date in mind -- how about we try for early next week (of April 15)? I am happy to do the backport PRs -- should we make a maintenance branch? Or maybe we should use https://github.com/apache/arrow-datafusion/tree/branch-37 🤔 ?
That sounds good. My plan was to use `branch-{version}` for any maintenance releases, following the pattern used by Apache Spark., so we would release any 37.x.x releases from branch-37.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2059991880
> Note that #9900 didn't in fact make all the udf's public - the string and unicode modules do not in fact have the modules changed to pub nor is default implemented for the functions that I spot checked.
Thank you -- moved the conversation to https://github.com/apache/arrow-datafusion/pull/10107#issuecomment-2059991319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2030486150
cc @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2049929018
Thanks @alamb. I have more bandwidth now to help support maintenance releases. Do you have a target date in mind for 37.1.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063962278
Release vote thread: https://lists.apache.org/thread/33bkbrlkqv962y0topx9rlqg19g5q2vk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/datafusion/issues/9904#issuecomment-2070863362
Filed https://github.com/apache/datafusion/issues/10181 to track the issue with `internal error: entered unreachable code: NamedStructField should be rewritten in OperatorToFunction`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [I] Release DataFusion `37.1.0` (non breaking API release) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9904:
URL: https://github.com/apache/arrow-datafusion/issues/9904#issuecomment-2063573890
I am now beginning the process of creating a release candidate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org