You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/02/26 11:45:31 UTC
[PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
alamb opened a new pull request, #14891:
URL: https://github.com/apache/datafusion/pull/14891
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Related to https://github.com/apache/datafusion/issues/14123
- Related to https://github.com/apache/datafusion/pull/14224
## Rationale for this change
As we have discussed in tickets, it would be good to have help users upgrade to new DataFusion versions, especially when major / disruptive API changes like https://github.com/apache/datafusion/pull/14224 are included
- Related to https://github.com/apache/datafusion/issues/13525
- Related to https://github.com/apache/datafusion/issues/13661
## What changes are included in this PR?
1. Add a upgrade guide section:
2. Add details on the upgrade to 46.0.0
## Are these changes tested?
I build the docs manually
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#issuecomment-2684755928
This will be definitely helpful for the user's upgrade, thank you @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on code in PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#discussion_r1971530350
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -0,0 +1,65 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Upgrade Guides
+
+## DataFusion `46.0.0`
+
+
+### Changes to `invoke()` and `invoke_batch()` deprecated
+
+We are migrating away from `ScalarUDFImpl::invoke()` and
Review Comment:
@alamb Yes, users may see an error similar to something like this (depending on which `UDF` they are calling `.invoke()` or `.invoke_batch()` with):
```
This feature is not implemented: Function concat does not implement invoke but called
```
Here is an example of resolving the error:
**Old Code:**
```
impl ScalarUDFImpl for SparkConcat {
...
fn invoke_batch(&self, args: &[ColumnarValue], number_rows: usize) -> Result<ColumnarValue> {
if args
.iter()
.any(|arg| matches!(arg.data_type(), DataType::List(_)))
{
ArrayConcat::new().invoke_batch(args, number_rows)
} else {
ConcatFunc::new().invoke_batch(args, number_rows)
}
}
}
```
**New Code:**
```
impl ScalarUDFImpl for SparkConcat {
...
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
if args
.args
.iter()
.any(|arg| matches!(arg.data_type(), DataType::List(_)))
{
ArrayConcat::new().invoke_with_args(args)
} else {
ConcatFunc::new().invoke_with_args(args)
}
}
}
```
Changes:
**1.**
From:
`fn invoke_batch(&self, args: &[ColumnarValue], number_rows: usize)` OR
`fn invoke(&self, args: &[ColumnarValue])`
To:
`fn invoke_with_args(&self, args: ScalarFunctionArgs)`
**2.**
From:
`.invoke_batch(args, number_rows)` OR
`.invoke(args)`
To:
`.invoke_with_args(args)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "shehabgamin (via GitHub)" <gi...@apache.org>.
shehabgamin commented on code in PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#discussion_r1975425387
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -0,0 +1,65 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Upgrade Guides
+
+## DataFusion `46.0.0`
+
+
+### Changes to `invoke()` and `invoke_batch()` deprecated
+
+We are migrating away from `ScalarUDFImpl::invoke()` and
Review Comment:
Woohoo!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #14891:
URL: https://github.com/apache/datafusion/pull/14891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#discussion_r1971452699
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -0,0 +1,65 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Upgrade Guides
+
+## DataFusion `46.0.0`
+
+
+### Changes to `invoke()` and `invoke_batch()` deprecated
+
+We are migrating away from `ScalarUDFImpl::invoke()` and
Review Comment:
I added this section based on the report from @shehabgamin in https://github.com/apache/datafusion/issues/14123#issuecomment-2684612667
@shehabgamin any chance you have an example of the kind of error you saw?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#issuecomment-2690695142
> Thanks @alamb for proceeding with this. Imho we still need to modify contribution guide to let contributors to maintain the upgration guide. Leaving this to one person is might be a huge burden
Thanks @comphead -- this is a good suggestion. I tried to do so in this PR with the following commit:
- 2c4a33b60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#issuecomment-2685510613
Thanks @alamb for proceeding with this. Imho we still need to modify contribution guide to let contributors to maintain the upgration guide. Leaving this to one person is might be a huge burden
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#issuecomment-2685607561
> > Thanks @alamb for proceeding with this. Imho we still need to modify contribution guide to let contributors to maintain the upgration guide. Leaving this to one person is might be a huge burden
>
> Sounds good . Maybe once we have an example of what an upgrade guide looks like it will be easier to add / point people to an example
>
> In general trying to change people's behavior with documentation is a bit diecy in my opinion
Yeah, example is awesome. The upgrade guide would be totally voluntarily but if anyone feels their changes are breaking and wanna give a downstream user a hint how to migrate they can refer to the example as you propose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#discussion_r1975421503
##########
docs/source/library-user-guide/upgrading.md:
##########
@@ -0,0 +1,65 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Upgrade Guides
+
+## DataFusion `46.0.0`
+
+
+### Changes to `invoke()` and `invoke_batch()` deprecated
+
+We are migrating away from `ScalarUDFImpl::invoke()` and
Review Comment:
Thank you so much @shehabgamin -- I have incorporated this content in 9c658c739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#issuecomment-2694061520
Thank you @xudong963 for the approval and @comphead for the review and @shehabgamin for the content assistance!
Since this PR affects only https://datafusion.apache.org/ it doesn't need to be backported to the branch-46 - now that I have merged it it it will show up on the website without having to be part of the release tarball
Maybe we could put a link to https://datafusion.apache.org/library-user-guide/upgrading.html in release notes / readme
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#issuecomment-2685558068
> Thanks @alamb for proceeding with this. Imho we still need to modify contribution guide to let contributors to maintain the upgration guide. Leaving this to one person is might be a huge burden
Sounds good . Maybe once we have an example of what an upgrade guide looks like it will be easier to add / point people to an example
In general trying to change people's behavior with documentation is a bit diecy in my opinion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add Upgrade Guide for DataFusion 46.0.0 [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #14891:
URL: https://github.com/apache/datafusion/pull/14891#issuecomment-2692102278
After the PR is merged, I'll update changlog PR and make the branch-46.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org