You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "kazdy (via GitHub)" <gi...@apache.org> on 2024/09/21 11:13:08 UTC
[PR] build(deps): Upgrade datafusion from 0.41 to 0.42 [hudi-rs]
kazdy opened a new pull request, #149:
URL: https://github.com/apache/hudi-rs/pull/149
## Description
<!--- Describe your changes in detail -->
<!--- If it fixes an open issue, please link to the issue here. -->
<!--- Please link any related issues and PRs as well. -->
## How are the changes test-covered
- [ ] N/A
- [ ] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #149:
URL: https://github.com/apache/hudi-rs/pull/149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770519805
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
"pyarrow-hotfix",
Review Comment:
pyarrow-hotfix is not needed since 0.14.1
I don't have a strong opinion on upgrading this, it seemed like a good idea, but I can revert this.
edit:
I checked some popular libraries that use pyarrow and declare a dependency:
datafusion: pyarrow>=11.0.0,
polars: pyarrow >= 7.0.0,
daft: pyarrow >= 7.0.0,
lance: pyarrow>=12
so should not cause issues for users (?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770598070
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
Review Comment:
```suggestion
"pyarrow>=11.0.0",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): Upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1769695299
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
"pyarrow-hotfix",
Review Comment:
this is the minimum required version for pyarrow, isn't making it at least 17 too aggressive? also can you check if pyarrow-hotfix still needed if we upgrade to a newer version here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#issuecomment-2366875673
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/149?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 87.82%. Comparing base [(`e23e6ed`)](https://app.codecov.io/gh/apache/hudi-rs/commit/e23e6ed3190e412a9d196fd3959b54c27ca75e2b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`94b0b69`)](https://app.codecov.io/gh/apache/hudi-rs/commit/94b0b69c19b4254be9ad45474af88f79ad45ab7c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #149 +/- ##
=======================================
Coverage 87.82% 87.82%
=======================================
Files 14 14
Lines 731 731
=======================================
Hits 642 642
Misses 89 89
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/hudi-rs/pull/149?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): Upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#issuecomment-2365149487
Datafusion plan changed a bit:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1797631700
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=11.0.0",
Review Comment:
we need to update in-code version check also, in `test_table_read.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): Upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770519805
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
"pyarrow-hotfix",
Review Comment:
pyarrow-hotfix is not needed since 0.14.1
I don't have a strong opinion on upgrading this, it seemed like a good idea, but I can revert this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770599363
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
"pyarrow-hotfix",
Review Comment:
sure, sounds good thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770519805
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
"pyarrow-hotfix",
Review Comment:
pyarrow-hotfix is not needed since 14.0.1
I don't have a strong opinion on upgrading this, it seemed like a good idea, but I can revert this.
edit:
I checked some popular libraries that use pyarrow and declare a dependency:
datafusion: pyarrow>=11.0.0,
polars: pyarrow >= 7.0.0,
daft: pyarrow >= 7.0.0,
lance: pyarrow>=12
so should not cause issues for users (?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770596221
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
"pyarrow-hotfix",
Review Comment:
thanks for the useful info. what i meant was: this would make hudi-rs pyarrow requirement too strict and exclude people to use when they have lower pyarrow version. I'd align with datafusion's version requirement since we have the integration and it's not too old or too new
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770597050
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
Review Comment:
```suggestion
"pyarrow>=11.0.0",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] build(deps): upgrade datafusion from 0.41 to 0.42 [hudi-rs]
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on code in PR #149:
URL: https://github.com/apache/hudi-rs/pull/149#discussion_r1770597050
##########
python/pyproject.toml:
##########
@@ -23,19 +23,18 @@ build-backend = "maturin"
name = "hudi"
description = "Native Python binding for Apache Hudi, based on hudi-rs."
urls = { repository = "https://github.com/apache/hudi-rs/tree/main/python/" }
-requires-python = ">=3.8"
+requires-python = ">=3.9"
keywords = ["apachehudi", "hudi", "datalake", "arrow"]
license = "Apache License 2.0"
classifiers = [
"License :: OSI Approved :: Apache Software License",
- "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12"
]
dependencies = [
- "pyarrow>=8",
+ "pyarrow>=17",
Review Comment:
```suggestion
"pyarrow>=11.0.0",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org