You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gorilla-seb (via GitHub)" <gi...@apache.org> on 2025/06/20 12:51:34 UTC
[I] Python package size has ballooned dramatically [iceberg-rust]
gorilla-seb opened a new issue, #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458
From 0.4.0 to 0.5.0 the size of `pyiceberg-core` Python packages has increased dramatically.
[Python wheels for 0.4.0](https://pypi.org/project/pyiceberg-core/0.4.0/#files): between ~ 0.7 and 1.5 MiB
[Python wheels for 0.5.1](https://pypi.org/project/pyiceberg-core/0.5.1/#files): 25 to 55MiB
This is reflected in the installed package (example from OSX, but Linux is similar):
```console
% du -sh .venv/lib/python3.12/site-packages/pyiceberg_core/*
4.0K .venv/lib/python3.12/site-packages/pyiceberg_core/__init__.py
4.0M .venv/lib/python3.12/site-packages/pyiceberg_core/pyiceberg_core_rust.abi3.so
% poetry update pyiceberg-core
Package operations: 0 installs, 1 update, 0 removals
- Updating pyiceberg-core (0.4.0 -> 0.5.1)
% du -sh .venv/lib/python3.12/site-packages/pyiceberg_core/*
4.0K .venv/lib/python3.12/site-packages/pyiceberg_core/__init__.py
162M .venv/lib/python3.12/site-packages/pyiceberg_core/pyiceberg_core_rust.abi3.so
```
In increase by over 4000% between minor versions seems like something is wrong with the build. Moreover, packages this large are borderline unusable on a service like AWS lambda where total size is limited to 250MiB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Python package size has ballooned dramatically [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458#issuecomment-3308165635
Closing this for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Python package size has ballooned dramatically [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #1458: Python package size has ballooned dramatically
URL: https://github.com/apache/iceberg-rust/issues/1458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Python package size has ballooned dramatically [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458#issuecomment-2993228270
hey @gorilla-seb i believe this is the result of #1324 when we added the datafusion table provider to the python binding.
> In increase by over 4000% between minor versions seems like something is wrong with the build
this is because pyiceberg_core only packaged the transform functions before, https://github.com/apache/iceberg-rust/blob/f1e79c0d9c76e90cb4ceb1f2c409ef8d1b68b273/bindings/python/src/lib.rs#L28. And those are pretty small
> Moreover, packages this large are borderline unusable on a service like AWS lambda where total size is limited to 250MiB.
pyiceberg_core is mainly used for the pyiceberg library since it doesnt have any standalone features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Python package size has ballooned dramatically [iceberg-rust]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458#issuecomment-3522787860
btw we reduce the size of the wheels significantly in #1839
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org