You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "eitsupi (via GitHub)" <gi...@apache.org> on 2025/05/31 04:18:56 UTC

[I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

eitsupi opened a new issue, #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764

   This may be a silly question, but when I considered the smallest Arrow dependency in Python, I was surprised to find that `arro3-core` has a smaller wheel size (2.5MB) than `nanoarrow` in Linux (4MB).
   I checked and it seems that the size of the `nanoarrow` wheel is several times larger on Linux (4MB) than it is for macOS and Windows (0.7MB).
   
   ![nanoarrow](https://github.com/user-attachments/assets/7f1ee1bd-ed8e-43ab-83f0-8f162fd1f61b)
   
   ![arro3-core](https://github.com/user-attachments/assets/e1c5a366-99f8-47eb-a460-c1853db05eb8)
   
   Is this intentional and can it be avoided?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2963156137

   Closing since I think the meson update fixes this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2925160587

   Without checking in detail I have 2 suspicions:
   - clang seems to produces smaller binaries on average
   - I think Linux wheels bundle more libraries than macros due to system heterogeneity 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2930934136

   That's a great point...I should have checked the nightlies/I forgot that we hadn't released since that change 🤦 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2925230151

   I think Cython is also an issue with respect to large binary sizes (the generated C code is absolutely wild). With some effort we might be able to get it smaller (or just use arro3, which is great too!). One of the things I think we'd like to do if any of us has the time is rewrite the bindings in nanobind (perhaps coupled with expanding the C++ capabilities of nanoarrow in the process). I don't forsee having time to do that in the near term but would be happy to review if anybody is interested!
   
   I did check to make sure that what we're looking at *is* binary size, and it appears to be the case (there was a previous issue a while back where I'd been including a bunch of source code which had inflated the size of the wheels considerably).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2930712823

   We can also consider building the pip wheels with `-Dbuildtype=minsize` instead of release, if we cared for size over speed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot closed issue #764: Python wheel size for Linux seems big (?)
URL: https://github.com/apache/arrow-nanoarrow/issues/764


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2930703623

   The latest version of Cython has some new functionality that both pandas and scikit-learn have used to reduce their wheel sizes. Might be worth trying this in the near term:
   
   https://github.com/pandas-dev/pandas/pull/61384


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2930868096

   After some more research I think this might be resolved on main already, by our switch to using meson-python from setuptools. 
   
   To illustrate, if I download the pypi wheel and check the top 10 file sizes, I see this:
   
   ```
   find . -type f -exec ls -alh {} \; | sort -hr -k5 | head 
   -rwxr-xr-x 1 willayd willayd 2.6M Oct  7  2024 ./_schema.cpython-311-x86_64-linux-gnu.so
   -rwxr-xr-x 1 willayd willayd 2.6M Oct  7  2024 ./_buffer.cpython-311-x86_64-linux-gnu.so
   -rwxr-xr-x 1 willayd willayd 2.5M Oct  7  2024 ./_array.cpython-311-x86_64-linux-gnu.so
   -rwxr-xr-x 1 willayd willayd 2.1M Oct  7  2024 ./_ipc_lib.cpython-311-x86_64-linux-gnu.so
   -rwxr-xr-x 1 willayd willayd 1.6M Oct  7  2024 ./_types.cpython-311-x86_64-linux-gnu.so
   -rwxr-xr-x 1 willayd willayd 1.5M Oct  7  2024 ./_array_stream.cpython-311-x86_64-linux-gnu.so
   -rwxr-xr-x 1 willayd willayd 926K Oct  7  2024 ./_device.cpython-311-x86_64-linux-gnu.so
   -rwxr-xr-x 1 willayd willayd 914K Oct  7  2024 ./_utils.cpython-311-x86_64-linux-gnu.so
   -rw-r--r-- 1 willayd willayd 39K Oct  7  2024 ./_buffer.pyx
   -rw-r--r-- 1 willayd willayd 37K Oct  7  2024 ./schema.py
   ```
   
   But with a local build my shared libraries clock in much smaller:
   
   ```
   -rwxrwxr-x 1 willayd willayd 695K Jun  2  2025 ./_ipc_lib.cpython-313-x86_64-linux-gnu.so
   -rwxrwxr-x 1 willayd willayd 387K Jun  2  2025 ./_array.cpython-313-x86_64-linux-gnu.so
   -rwxrwxr-x 1 willayd willayd 363K Jun  2  2025 ./_schema.cpython-313-x86_64-linux-gnu.so
   -rwxrwxr-x 1 willayd willayd 338K Jun  2  2025 ./_buffer.cpython-313-x86_64-linux-gnu.so
   -rwxrwxr-x 1 willayd willayd 249K Jun  2  2025 ./_array_stream.cpython-313-x86_64-linux-gnu.so
   -rwxrwxr-x 1 willayd willayd 178K Jun  2  2025 ./_device.cpython-313-x86_64-linux-gnu.so
   -rwxrwxr-x 1 willayd willayd 164K Jun  2  2025 ./_utils.cpython-313-x86_64-linux-gnu.so
   -rwxrwxr-x 1 willayd willayd 149K Jun  2  2025 ./_types.cpython-313-x86_64-linux-gnu.so
   -rw-r--r-- 1 willayd willayd 40K Jun  1 04:06 ./schema.py
   -rw-r--r-- 1 willayd willayd 23K Jun  1 04:06 ./c_array.py
   ```
   
   I don't think auditwheel would drastically change the size of those libraries, so I think we should monitor this as we go through building wheels in the pre-release process


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Python wheel size for Linux seems big (?) [arrow-nanoarrow]

Posted by "assignUser (via GitHub)" <gi...@apache.org>.
assignUser commented on issue #764:
URL: https://github.com/apache/arrow-nanoarrow/issues/764#issuecomment-2931147255

   Nice, good to know!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org