You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gorilla-seb (via GitHub)" <gi...@apache.org> on 2025/06/20 12:51:34 UTC

[I] Python package size has ballooned dramatically [iceberg-rust]

gorilla-seb opened a new issue, #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458

   From 0.4.0 to 0.5.0 the size of `pyiceberg-core` Python packages has increased dramatically.
   
   [Python wheels for 0.4.0](https://pypi.org/project/pyiceberg-core/0.4.0/#files): between ~ 0.7 and 1.5 MiB
   
   [Python wheels for 0.5.1](https://pypi.org/project/pyiceberg-core/0.5.1/#files): 25 to 55MiB
   
   This is reflected in the installed package (example from OSX, but Linux is similar): 
   ```console
   % du -sh .venv/lib/python3.12/site-packages/pyiceberg_core/*
   4.0K    .venv/lib/python3.12/site-packages/pyiceberg_core/__init__.py
   4.0M    .venv/lib/python3.12/site-packages/pyiceberg_core/pyiceberg_core_rust.abi3.so
   
   % poetry update pyiceberg-core
   Package operations: 0 installs, 1 update, 0 removals
   
     - Updating pyiceberg-core (0.4.0 -> 0.5.1)
   
   % du -sh .venv/lib/python3.12/site-packages/pyiceberg_core/*
   4.0K    .venv/lib/python3.12/site-packages/pyiceberg_core/__init__.py
   162M    .venv/lib/python3.12/site-packages/pyiceberg_core/pyiceberg_core_rust.abi3.so
   ```
   
   In increase by over 4000% between minor versions seems like something is wrong with the build. Moreover, packages this large are borderline unusable on a service like AWS lambda where total size is limited to 250MiB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Python package size has ballooned dramatically [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458#issuecomment-3308165635

   Closing this for now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Python package size has ballooned dramatically [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #1458: Python package size has ballooned dramatically
URL: https://github.com/apache/iceberg-rust/issues/1458


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Python package size has ballooned dramatically [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458#issuecomment-2993228270

   hey @gorilla-seb i believe this is the result of #1324 when we added the datafusion table provider to the python binding. 
   
   > In increase by over 4000% between minor versions seems like something is wrong with the build
   
   this is because pyiceberg_core only packaged the transform functions before, https://github.com/apache/iceberg-rust/blob/f1e79c0d9c76e90cb4ceb1f2c409ef8d1b68b273/bindings/python/src/lib.rs#L28. And those are pretty small
   
   > Moreover, packages this large are borderline unusable on a service like AWS lambda where total size is limited to 250MiB.
   
   pyiceberg_core is mainly used for the pyiceberg library since it doesnt have any standalone features. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Python package size has ballooned dramatically [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1458:
URL: https://github.com/apache/iceberg-rust/issues/1458#issuecomment-3522787860

   btw we reduce the size of the wheels significantly in #1839 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org