You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "geruh (via GitHub)" <gi...@apache.org> on 2025/10/10 17:30:37 UTC
[PR] Add support for UV dependency management [iceberg-python]
geruh opened a new pull request, #2601:
URL: https://github.com/apache/iceberg-python/pull/2601
Closes #2553
# Rationale for this change
During the release process cleanup, I also had prototyped a migration from Poetry to UV. I used the https://github.com/mkniewallner/migrate-to-uv package to migrate to UV and ran a release workflow and some others like doc serve for validation.
This PR also migrates the Makefile to use UV. I'll still need to upgrade the docs to reflect the UV contribution workflow. I think it would be a good start to push up as a prototype, and have others checkout and give it a try.
My testing below.
## Are these changes tested?
Yes
- [x] `make install` Environment setup and dependency installation
- [x] `make test` - Unit tests
- [x] `make test-integration` - Integration tests with all extras
- [x] `make lint` - Linting and code quality checks
- [x] `make doc-install && make doc-serve` - Documentation
## Are there any user-facing changes?
No user facing changes. Contributors will need to use UV instead of Poetry for development, but all make targets remain the same. So not really?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#issuecomment-3449017678
i cant seem to get `cibuildwheel` to work on github ci 🤔
```
Error: cibuildwheel:
Build failed because a pure Python wheel was generated.
If you intend to build a pure-Python wheel, you don't need
cibuildwheel - use `pip wheel .`, `pipx run build --wheel`, `uv
build --wheel`, etc. instead. You only need cibuildwheel if you
have compiled (not Python) code in your wheels making them depend
on the platform.
If you expected a platform wheel, check your project configuration,
or run cibuildwheel with CIBW_BUILD_VERBOSITY=1 to view build logs.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2464137001
##########
pyproject.toml:
##########
@@ -14,111 +14,163 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
Review Comment:
```suggestion
requires-python = ">=3.10,<4.0.0"
```
we just removed 3.9 in #2554
##########
.github/workflows/svn-build-artifacts.yml:
##########
@@ -47,13 +47,13 @@ jobs:
3.11
3.12
- - name: Install poetry
- run: make install-poetry
+ - name: Install UV
+ run: make install-uv
Review Comment:
i think we need to use
```
- name: Install uv
uses: astral-sh/setup-uv@v6
```
instead of `make install-uv`
When i was testing "Nightly PyPI Build", `make install-uv` fails for windows machine with
```
Run uv version "0.10.0.dev20251026230001"
uv: D:\a\_temp\c958d63c-abd9-4fae-8896-0d96857f661f.ps1:2
Line |
2 | uv version "0.10.0.dev20251026230001"
| ~~
| The term 'uv' is not recognized as a name of a cmdlet, function, script file, or executable program. Check the
| spelling of the name, or if a path was included, verify that the path is correct and try again.
Error: Process completed with exit code 1.
```
https://github.com/kevinjqliu/iceberg-python/actions/runs/18824958012/job/53706130681
##########
pyproject.toml:
##########
@@ -14,111 +14,163 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
+readme = "README.md"
+license = "Apache-2.0"
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.9",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "License :: OSI Approved :: Apache Software License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.9",
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#issuecomment-3514831073
Thanks for the PR @geruh, super exciting to start using uv. Thanks for the review @Fokko @rambleraptor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2508487383
##########
MANIFEST.in:
##########
@@ -15,4 +15,18 @@
# specific language governing permissions and limitations
# under the License.
-graft src
+# Include Cython source files for building from source
+recursive-include pyiceberg *.pyx *.c
Review Comment:
`decoder_fast.c` is compiled when creating the sdist. I can prune this but, we still need to include `decoder_basic.c` and `decoder_fast.pyx`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2424857410
##########
pyproject.toml:
##########
@@ -14,111 +14,163 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
+readme = "README.md"
+license = "Apache-2.0"
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.9",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "License :: OSI Approved :: Apache Software License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.9",
Review Comment:
Nice one, maybe we should add `pretty-format-toml ` to pre-commit to format the `toml` files 👍
##########
pyproject.toml:
##########
@@ -14,111 +14,163 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
+readme = "README.md"
+license = "Apache-2.0"
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.9",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "License :: OSI Approved :: Apache Software License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
]
-packages = [
- { include = "pyiceberg" },
- { from = "vendor", include = "fb303" },
- { from = "vendor", include = "hive_metastore" },
- { include = "tests", format = "sdist" },
- { include = "Makefile", format = "sdist" },
- { include = "NOTICE", format = [
- "sdist",
- "wheel",
- ] },
+dependencies = [
+ "mmh3>=4.0.0,<6.0.0",
+ "requests>=2.20.0,<3.0.0",
+ "click>=7.1.1,<9.0.0",
+ "rich>=10.11.0,<15.0.0",
+ "strictyaml>=1.7.0,<2.0.0", # CVE-2020-14343 was fixed in 5.4.
+ "pydantic>=2.0,!=2.4.0,!=2.4.1,<2.12.0", # 2.4.0, 2.4.1, 2.12.0 has a critical bug
+ "sortedcontainers==2.4.0",
+ "fsspec>=2023.1.0",
+ "pyparsing>=3.1.0,<4.0.0",
+ "tenacity>=8.2.3,<10.0.0",
+ "pyroaring>=1.0.0,<2.0.0",
+ "cachetools>=5.5,<7.0",
+ "zstandard>=0.13.0,<1.0.0"
]
-include = [
- { path = "dev", format = "sdist" },
- { path = "pyiceberg/**/*.so", format = "wheel" },
- { path = "pyiceberg/**/*.pyd", format = "wheel" },
- { path = "poetry.lock", format = "sdist" },
+
+[project.optional-dependencies]
+pyarrow = [
+ "pyarrow>=17.0.0",
Review Comment:
nit, maybe find a way to deduplicate the constraints. `pyarrow` is mentioned multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2508519258
##########
pyproject.toml:
##########
@@ -14,147 +14,133 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.10.0,<4.0.0"
+readme = "README.md"
+license = "Apache-2.0"
+license-files = ["LICEN[CS]E*", "NOTICE*"]
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+ "mmh3>=4.0.0,<6.0.0",
+ "requests>=2.20.0,<3.0.0",
+ "click>=7.1.1,<9.0.0",
+ "rich>=10.11.0,<15.0.0",
+ "strictyaml>=1.7.0,<2.0.0", # CVE-2020-14343 was fixed in 5.4.
+ "pydantic>=2.0,<3.0,!=2.4.0,!=2.4.1,!=2.12.0,!=2.12.1", # 2.4.0, 2.4.1, 2.12.0, 2.12.1 has a critical bug
+ "sortedcontainers==2.4.0",
+ "fsspec>=2023.1.0",
+ "pyparsing>=3.1.0,<4.0.0",
+ "tenacity>=8.2.3,<10.0.0",
+ "pyroaring>=1.0.0,<2.0.0",
+ "cachetools>=5.5,<7.0",
+ "zstandard>=0.13.0,<1.0.0"
+]
+
+[project.optional-dependencies]
+pyarrow = [
+ "pyarrow>=17.0.0",
+ "pyiceberg-core>=0.5.1,<0.8.0",
+]
+pandas = [
+ "pyiceberg[pyarrow]",
+ "pandas>=1.0.0,<3.0.0",
+]
+duckdb = [
+ "pyiceberg[pyarrow]",
+ "duckdb>=0.5.0,<2.0.0",
+]
+ray = [
+ "pyiceberg[pyarrow]",
Review Comment:
These will all be auto updated together, but the `"pyiceberg[pyarrow]"` dependencies like pandas and duckdb also needed pyiceberg-core.
> uv will edit all the instances at once
I agree with this sentiment though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2508482132
##########
MANIFEST.in:
##########
@@ -15,4 +15,18 @@
# specific language governing permissions and limitations
# under the License.
-graft src
+# Include Cython source files for building from source
+recursive-include pyiceberg *.pyx *.c
+
+# Include test files in sdist
+recursive-include tests *.py
+
+# Include development files
+include Makefile
+recursive-include dev *
+
+# Exclude build artifacts
+prune .venv
+prune build
+prune dist
+prune .pytest_cache
Review Comment:
sadly these are auto generated after manifest.in is used in setuptools
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2437786132
##########
pyproject.toml:
##########
@@ -14,111 +14,163 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
+readme = "README.md"
+license = "Apache-2.0"
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.9",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "License :: OSI Approved :: Apache Software License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
]
-packages = [
- { include = "pyiceberg" },
- { from = "vendor", include = "fb303" },
- { from = "vendor", include = "hive_metastore" },
- { include = "tests", format = "sdist" },
- { include = "Makefile", format = "sdist" },
- { include = "NOTICE", format = [
- "sdist",
- "wheel",
- ] },
+dependencies = [
+ "mmh3>=4.0.0,<6.0.0",
+ "requests>=2.20.0,<3.0.0",
+ "click>=7.1.1,<9.0.0",
+ "rich>=10.11.0,<15.0.0",
+ "strictyaml>=1.7.0,<2.0.0", # CVE-2020-14343 was fixed in 5.4.
+ "pydantic>=2.0,!=2.4.0,!=2.4.1,<2.12.0", # 2.4.0, 2.4.1, 2.12.0 has a critical bug
+ "sortedcontainers==2.4.0",
+ "fsspec>=2023.1.0",
+ "pyparsing>=3.1.0,<4.0.0",
+ "tenacity>=8.2.3,<10.0.0",
+ "pyroaring>=1.0.0,<2.0.0",
+ "cachetools>=5.5,<7.0",
+ "zstandard>=0.13.0,<1.0.0"
]
-include = [
- { path = "dev", format = "sdist" },
- { path = "pyiceberg/**/*.so", format = "wheel" },
- { path = "pyiceberg/**/*.pyd", format = "wheel" },
- { path = "poetry.lock", format = "sdist" },
+
+[project.optional-dependencies]
+pyarrow = [
+ "pyarrow>=17.0.0",
Review Comment:
Sounds good! it looks like poetry had a way of doing this before with the `[tool.poetry.extras]`, kind of acting as a version catalog. UV doesn't have support for this but respects the PEP standard for project dependencies. In this case one way we can de-duplicate is by using the PEP 508 Extras grammar which allows for self referencing other dependencies in the project like:
```
[project.optional-dependencies]
pyarrow = [
"pyarrow>=17.0.0",
"pyiceberg-core>=0.5.1,<0.7.0",
]
pandas = [
"pyiceberg[pyarrow]",
"pandas>=1.0.0,<3.0.0",
]
```
What do you think @Fokko?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2508338735
##########
pyproject.toml:
##########
@@ -14,147 +14,133 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.10.0,<4.0.0"
+readme = "README.md"
+license = "Apache-2.0"
+license-files = ["LICEN[CS]E*", "NOTICE*"]
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+ "mmh3>=4.0.0,<6.0.0",
+ "requests>=2.20.0,<3.0.0",
+ "click>=7.1.1,<9.0.0",
+ "rich>=10.11.0,<15.0.0",
+ "strictyaml>=1.7.0,<2.0.0", # CVE-2020-14343 was fixed in 5.4.
+ "pydantic>=2.0,<3.0,!=2.4.0,!=2.4.1,!=2.12.0,!=2.12.1", # 2.4.0, 2.4.1, 2.12.0, 2.12.1 has a critical bug
+ "sortedcontainers==2.4.0",
+ "fsspec>=2023.1.0",
+ "pyparsing>=3.1.0,<4.0.0",
+ "tenacity>=8.2.3,<10.0.0",
+ "pyroaring>=1.0.0,<2.0.0",
+ "cachetools>=5.5,<7.0",
+ "zstandard>=0.13.0,<1.0.0"
+]
+
+[project.optional-dependencies]
+pyarrow = [
+ "pyarrow>=17.0.0",
+ "pyiceberg-core>=0.5.1,<0.8.0",
+]
+pandas = [
+ "pyiceberg[pyarrow]",
+ "pandas>=1.0.0,<3.0.0",
+]
+duckdb = [
+ "pyiceberg[pyarrow]",
+ "duckdb>=0.5.0,<2.0.0",
+]
+ray = [
+ "pyiceberg[pyarrow]",
+ "ray>=2.10.0,<=2.44.0",
+ "pandas>=1.0.0,<3.0.0",
]
-packages = [
- { include = "pyiceberg" },
- { from = "vendor", include = "fb303" },
- { from = "vendor", include = "hive_metastore" },
- { include = "tests", format = "sdist" },
- { include = "Makefile", format = "sdist" },
- { include = "NOTICE", format = [
- "sdist",
- "wheel",
- ] },
+bodo = ["bodo>=2025.7.4"]
+daft = ["daft>=0.5.0"]
+polars = ["polars>=1.21.0,<2"]
+snappy = ["python-snappy>=0.6.0,<1.0.0"]
+hive = ["thrift>=0.13.0,<1.0.0"]
+hive-kerberos = [
+ "thrift>=0.13.0,<1.0.0",
+ "thrift-sasl>=0.4.3",
+ "kerberos>=1.3.1,<2",
]
-include = [
- { path = "dev", format = "sdist" },
- { path = "pyiceberg/**/*.so", format = "wheel" },
- { path = "pyiceberg/**/*.pyd", format = "wheel" },
- { path = "poetry.lock", format = "sdist" },
+s3fs = ["s3fs>=2023.1.0"]
+glue = ["boto3>=1.24.59"]
+adlfs = ["adlfs>=2024.7.0"]
+dynamodb = ["boto3>=1.24.59"]
+bigquery = ["google-cloud-bigquery>=3.33.0,<4"]
+sql-postgres = [
+ "sqlalchemy>=2.0.18,<3",
+ "psycopg2-binary>=2.9.6",
]
+sql-sqlite = ["sqlalchemy>=2.0.18,<3"]
+gcsfs = ["gcsfs>=2023.1.0"]
+rest-sigv4 = ["boto3>=1.24.59"]
+hf = ["huggingface-hub>=0.24.0"]
+pyiceberg-core = ["pyiceberg-core>=0.5.1,<0.7.0"]
Review Comment:
```suggestion
pyiceberg-core = ["pyiceberg-core>=0.5.1,<0.8.0"]
```
##########
dev/.rat-excludes:
##########
@@ -3,5 +3,9 @@
build
.git
.gitignore
-poetry.lock
+uv.lock
mkdocs/*
+setup.cfg
+(^|.*/)[^/]*\.egg-info(/.*)?$
+# Cython generated files
+decoder_fast.c
Review Comment:
nit: should we exclude `setup.cfg` and `pyiceberg.egg-info` these from sdist instead?
`decoder_fast.c` has the apache copyright headers, so we can safely remove from this list
##########
pyproject.toml:
##########
@@ -14,147 +14,133 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.10.0,<4.0.0"
+readme = "README.md"
+license = "Apache-2.0"
+license-files = ["LICEN[CS]E*", "NOTICE*"]
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+ "mmh3>=4.0.0,<6.0.0",
+ "requests>=2.20.0,<3.0.0",
+ "click>=7.1.1,<9.0.0",
+ "rich>=10.11.0,<15.0.0",
+ "strictyaml>=1.7.0,<2.0.0", # CVE-2020-14343 was fixed in 5.4.
+ "pydantic>=2.0,<3.0,!=2.4.0,!=2.4.1,!=2.12.0,!=2.12.1", # 2.4.0, 2.4.1, 2.12.0, 2.12.1 has a critical bug
+ "sortedcontainers==2.4.0",
+ "fsspec>=2023.1.0",
+ "pyparsing>=3.1.0,<4.0.0",
+ "tenacity>=8.2.3,<10.0.0",
+ "pyroaring>=1.0.0,<2.0.0",
+ "cachetools>=5.5,<7.0",
+ "zstandard>=0.13.0,<1.0.0"
+]
+
+[project.optional-dependencies]
+pyarrow = [
+ "pyarrow>=17.0.0",
+ "pyiceberg-core>=0.5.1,<0.8.0",
+]
+pandas = [
+ "pyiceberg[pyarrow]",
+ "pandas>=1.0.0,<3.0.0",
+]
+duckdb = [
+ "pyiceberg[pyarrow]",
+ "duckdb>=0.5.0,<2.0.0",
+]
+ray = [
+ "pyiceberg[pyarrow]",
Review Comment:
nit: i think `pyiceberg[pyarrow]` here will install `pyiceberg-core` as well.
For correctness, i think it might be better to go back to using `pyarrow>=17.0.0`. Even with the duplication, i think uv will edit all the instances at once
##########
pyproject.toml:
##########
@@ -14,147 +14,133 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.10.0,<4.0.0"
+readme = "README.md"
+license = "Apache-2.0"
+license-files = ["LICEN[CS]E*", "NOTICE*"]
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+]
+dependencies = [
+ "mmh3>=4.0.0,<6.0.0",
+ "requests>=2.20.0,<3.0.0",
+ "click>=7.1.1,<9.0.0",
+ "rich>=10.11.0,<15.0.0",
+ "strictyaml>=1.7.0,<2.0.0", # CVE-2020-14343 was fixed in 5.4.
+ "pydantic>=2.0,<3.0,!=2.4.0,!=2.4.1,!=2.12.0,!=2.12.1", # 2.4.0, 2.4.1, 2.12.0, 2.12.1 has a critical bug
+ "sortedcontainers==2.4.0",
+ "fsspec>=2023.1.0",
+ "pyparsing>=3.1.0,<4.0.0",
+ "tenacity>=8.2.3,<10.0.0",
+ "pyroaring>=1.0.0,<2.0.0",
+ "cachetools>=5.5,<7.0",
+ "zstandard>=0.13.0,<1.0.0"
+]
+
+[project.optional-dependencies]
+pyarrow = [
+ "pyarrow>=17.0.0",
+ "pyiceberg-core>=0.5.1,<0.8.0",
+]
+pandas = [
+ "pyiceberg[pyarrow]",
+ "pandas>=1.0.0,<3.0.0",
+]
+duckdb = [
+ "pyiceberg[pyarrow]",
+ "duckdb>=0.5.0,<2.0.0",
+]
+ray = [
+ "pyiceberg[pyarrow]",
+ "ray>=2.10.0,<=2.44.0",
+ "pandas>=1.0.0,<3.0.0",
]
-packages = [
- { include = "pyiceberg" },
- { from = "vendor", include = "fb303" },
- { from = "vendor", include = "hive_metastore" },
- { include = "tests", format = "sdist" },
- { include = "Makefile", format = "sdist" },
- { include = "NOTICE", format = [
- "sdist",
- "wheel",
- ] },
+bodo = ["bodo>=2025.7.4"]
+daft = ["daft>=0.5.0"]
+polars = ["polars>=1.21.0,<2"]
+snappy = ["python-snappy>=0.6.0,<1.0.0"]
+hive = ["thrift>=0.13.0,<1.0.0"]
+hive-kerberos = [
+ "thrift>=0.13.0,<1.0.0",
+ "thrift-sasl>=0.4.3",
+ "kerberos>=1.3.1,<2",
]
-include = [
- { path = "dev", format = "sdist" },
- { path = "pyiceberg/**/*.so", format = "wheel" },
- { path = "pyiceberg/**/*.pyd", format = "wheel" },
- { path = "poetry.lock", format = "sdist" },
+s3fs = ["s3fs>=2023.1.0"]
+glue = ["boto3>=1.24.59"]
+adlfs = ["adlfs>=2024.7.0"]
+dynamodb = ["boto3>=1.24.59"]
+bigquery = ["google-cloud-bigquery>=3.33.0,<4"]
+sql-postgres = [
+ "sqlalchemy>=2.0.18,<3",
+ "psycopg2-binary>=2.9.6",
]
+sql-sqlite = ["sqlalchemy>=2.0.18,<3"]
+gcsfs = ["gcsfs>=2023.1.0"]
+rest-sigv4 = ["boto3>=1.24.59"]
+hf = ["huggingface-hub>=0.24.0"]
+pyiceberg-core = ["pyiceberg-core>=0.5.1,<0.7.0"]
Review Comment:
also need to update the lock file
```
uv lock --upgrade-package pyiceberg-core==0.7.0
```
##########
MANIFEST.in:
##########
@@ -15,4 +15,18 @@
# specific language governing permissions and limitations
# under the License.
-graft src
+# Include Cython source files for building from source
+recursive-include pyiceberg *.pyx *.c
+
+# Include test files in sdist
+recursive-include tests *.py
Review Comment:
diff between nightly and this branch shows `tests/table/bitmaps/*` files are missing, so maybe we should include everything in tests/
````
Only in nightly/pyiceberg-0.10.0.dev20251109003122/tests/table: bitmaps
```
```suggestion
recursive-include tests
```
##########
MANIFEST.in:
##########
@@ -15,4 +15,18 @@
# specific language governing permissions and limitations
# under the License.
-graft src
+# Include Cython source files for building from source
+recursive-include pyiceberg *.pyx *.c
+
+# Include test files in sdist
+recursive-include tests *.py
+
+# Include development files
+include Makefile
+recursive-include dev *
+
+# Exclude build artifacts
+prune .venv
+prune build
+prune dist
+prune .pytest_cache
Review Comment:
```suggestion
prune .pytest_cache
prune pyiceberg.egg-info
prune setup.cfg
```
can we get rid of these `pyiceberg.egg-info` and `setup.cfg` from sdist? These are auto generated and we can also remove them from `dev/.rat-excludes`
##########
MANIFEST.in:
##########
@@ -15,4 +15,18 @@
# specific language governing permissions and limitations
# under the License.
-graft src
+# Include Cython source files for building from source
+recursive-include pyiceberg *.pyx *.c
Review Comment:
diff between nightly and this branch shows `decoder_fast.c` is newly added. i think we can get rid of `*.pyx *.c`
```suggestion
recursive-include pyiceberg
```
##########
pyproject.toml:
##########
@@ -14,147 +14,133 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.10.0,<4.0.0"
+readme = "README.md"
+license = "Apache-2.0"
+license-files = ["LICEN[CS]E*", "NOTICE*"]
Review Comment:
```suggestion
license-files = ["LICENSE", "NOTICE"]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2500881417
##########
.github/workflows/pypi-build-artifacts.yml:
##########
@@ -70,13 +70,15 @@ jobs:
CIBW_ARCHS: "auto64"
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10"
CIBW_TEST_REQUIRES: "pytest==7.4.2 moto==5.0.1"
- CIBW_TEST_COMMAND: "pytest {project}/tests/avro/test_decoder.py"
+ CIBW_TEST_COMMAND: "PYTHON_GIL=1 pytest {project}/tests/avro/test_decoder.py"
+ CIBW_BUILD_VERBOSITY: 1
Review Comment:
```suggestion
```
nit: i dont think we need this
##########
pyproject.toml:
##########
@@ -291,43 +317,6 @@ ignore_missing_imports = true
module = "google.*"
ignore_missing_imports = true
-[tool.poetry.scripts]
-pyiceberg = "pyiceberg.cli.console:run"
-
-[build-system]
-requires = ["poetry-core>=1.0.0", "wheel", "Cython>=3.0.0", "setuptools"]
-build-backend = "poetry.core.masonry.api"
-
-[tool.poetry.build]
-generate-setup-file = false
-script = "build-module.py"
-
-[tool.poetry.extras]
-pyarrow = ["pyarrow", "pyiceberg-core"]
-pandas = ["pandas", "pyarrow"]
-duckdb = ["duckdb", "pyarrow"]
-ray = ["ray", "pyarrow", "pandas"]
-bodo = ["bodo"]
-daft = ["daft"]
-polars = ["polars"]
-snappy = ["python-snappy"]
-hive = ["thrift"]
-hive-kerberos = ["thrift", "thrift_sasl", "kerberos"]
-s3fs = ["s3fs"]
-glue = ["boto3"]
-adlfs = ["adlfs"]
-dynamodb = ["boto3"]
-bigquery = ["google-cloud-bigquery"]
-zstandard = ["zstandard"]
-sql-postgres = ["sqlalchemy", "psycopg2-binary"]
-sql-sqlite = ["sqlalchemy"]
-gcsfs = ["gcsfs"]
-rest-sigv4 = ["boto3"]
-hf = ["huggingface-hub"]
-pyiceberg-core = ["pyiceberg-core"]
-datafusion = ["datafusion"]
-gcp-auth = ["google-auth"]
-
[tool.pytest.ini_options]
testpaths = ["tests"]
Review Comment:
```suggestion
testpaths = ["tests"]
pythonpath = ["."]
```
had to add this for `make test` to work locally
##########
.github/workflows/pypi-build-artifacts.yml:
##########
@@ -70,13 +70,15 @@ jobs:
CIBW_ARCHS: "auto64"
CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10"
CIBW_TEST_REQUIRES: "pytest==7.4.2 moto==5.0.1"
- CIBW_TEST_COMMAND: "pytest {project}/tests/avro/test_decoder.py"
+ CIBW_TEST_COMMAND: "PYTHON_GIL=1 pytest {project}/tests/avro/test_decoder.py"
Review Comment:
```suggestion
CIBW_TEST_COMMAND: "pytest {project}/tests/avro/test_decoder.py"
```
do we need this? since we added `CIBW_SKIP: "cp3*t-*"` below
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2512572520
##########
pyproject.toml:
##########
@@ -14,147 +14,133 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.10.0,<4.0.0"
+readme = "README.md"
+license = "Apache-2.0"
+license-files = ["LICEN[CS]E*", "NOTICE*"]
Review Comment:
Updated this originally this came from how pep defined it
https://peps.python.org/pep-0639/#add-license-files-key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add support for UV dependency management [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #2601:
URL: https://github.com/apache/iceberg-python/pull/2601#discussion_r2425025287
##########
pyproject.toml:
##########
@@ -14,111 +14,163 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-[tool.poetry]
+[project]
name = "pyiceberg"
version = "0.10.0"
-readme = "README.md"
-homepage = "https://py.iceberg.apache.org/"
-repository = "https://github.com/apache/iceberg-python"
description = "Apache Iceberg is an open table format for huge analytic datasets"
-authors = ["Apache Software Foundation <de...@iceberg.apache.org>"]
-license = "Apache License 2.0"
+authors = [{ name = "Apache Software Foundation", email = "dev@iceberg.apache.org" }]
+requires-python = ">=3.9.2,<4.0.0,!=3.9.7"
+readme = "README.md"
+license = "Apache-2.0"
classifiers = [
- "License :: OSI Approved :: Apache Software License",
- "Operating System :: OS Independent",
- "Programming Language :: Python :: 3.9",
- "Programming Language :: Python :: 3.10",
- "Programming Language :: Python :: 3.11",
- "Programming Language :: Python :: 3.12",
+ "License :: OSI Approved :: Apache Software License",
+ "Operating System :: OS Independent",
+ "Programming Language :: Python :: 3.9",
Review Comment:
That's a great idea. #2605 if you want to merge it in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org