You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aschreiber1 (via GitHub)" <gi...@apache.org> on 2024/08/22 18:29:13 UTC
[I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
aschreiber1 opened a new issue, #1093:
URL: https://github.com/apache/iceberg-python/issues/1093
### Feature Request / Improvement
when you install pyiceberg via conda you get warnings like:
/home/coder/.conda/envs/coder/lib/python3.10/site-packages/pyiceberg/avro/decoder.py:185: UserWarning: Falling back to pure Python Avro decoder, missing Cython implementation
warnings.warn("Falling back to pure Python Avro decoder, missing Cython implementation")
Which means it is missing the fast avro decoder. It would be great to have this functionality to speed up our queries!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2745930649
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "JanKrl (via GitHub)" <gi...@apache.org>.
JanKrl commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2343901577
> Adding some context to the Avro decoder build process.
>
> We use Poetry to build the Avro decoder via this script
>
> https://github.com/apache/iceberg-python/blob/b5933756b5b488ec51cd56d5984731b6cc347f2b/pyproject.toml#L583
>
> https://github.com/apache/iceberg-python/blob/b5933756b5b488ec51cd56d5984731b6cc347f2b/build-module.py
>
> you can manually trigger by
>
> ```
> poetry build
> ```
>
> Depending on the platform, there might be some missing piece that's not allowing the build to succeed.
>
> @JanKrl can you try the above command and paste the output here for debugging?
Deleting python environment (env) solved the problem for me.
So, (un)fortunately I'm not able to reproduce it anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "fcrimins (via GitHub)" <gi...@apache.org>.
fcrimins commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2369589615
I must have missed the "Falling back to pure Python Avro decoder, missing Cython implementation" warning during installation, but I was indeed missing Cython, which was causing a different `decoder_fast` related error during `make test`:
```bash
~/iceberg-python$ make test
poetry run pytest tests/ -m "(unmarked or parametrize) and not integration"
=========================================================== test session starts ============================================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0
rootdir: /mnt/c/Users/.../iceberg-python
configfile: pyproject.toml
plugins: mock-3.14.0, checkdocs-2.10.1, requests-mock-1.12.1, lazy-fixture-0.6.3
collected 3588 items / 2 errors / 865 deselected / 2723 selected
================================================================== ERRORS ==================================================================
_______________________________________________ ERROR collecting tests/avro/test_decoder.py ________________________________________________
ImportError while importing test module '/mnt/c/Users/.../iceberg-python/tests/avro/test_decoder.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/avro/test_decoder.py:29: in <module>
from pyiceberg.avro.decoder_fast import CythonBinaryDecoder
E ModuleNotFoundError: No module named 'pyiceberg.avro.decoder_fast'
________________________________________________ ERROR collecting tests/avro/test_reader.py ________________________________________________
ImportError while importing test module '/mnt/c/Users/.../iceberg-python/tests/avro/test_reader.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/avro/test_reader.py:24: in <module>
from pyiceberg.avro.decoder_fast import CythonBinaryDecoder
E ModuleNotFoundError: No module named 'pyiceberg.avro.decoder_fast'
========================================================= short test summary info ==========================================================
ERROR tests/avro/test_decoder.py
ERROR tests/avro/test_reader.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================== 865 deselected, 2 errors in 23.91s ====================================================
make: *** [Makefile:42: test] Error 2
```
I was able to fix the test errors by following the Cython installation instructions documented [here](https://cython.readthedocs.io/en/latest/src/quickstart/install.html) and then rebuilding.
```bash
~/iceberg-python$ sudo apt-get install build-essential python3-dev
~/iceberg-python$ pip install --user Cython
~/iceberg-python$ poetry build
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "JanKrl (via GitHub)" <gi...@apache.org>.
JanKrl commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2340343134
I experience the same issue with installation via pypi (Windows):
`D:\code\...\.venv\Lib\site-packages\pyiceberg\avro\decoder.py:185: UserWarning: Falling back to pure Python Avro decoder, missing Cython implementation`
https://github.com/apache/iceberg-python/blob/9b9ed534b2022cb9a687f4ed876fadcc2457b31b/pyiceberg/avro/decoder.py#L177-L187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2344154285
Great to hear. did you clean the env manually or use `make clean`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2341456013
Adding some context to the Avro decoder build process.
We use Poetry to build the Avro decoder via this script
https://github.com/apache/iceberg-python/blob/b5933756b5b488ec51cd56d5984731b6cc347f2b/pyproject.toml#L583
https://github.com/apache/iceberg-python/blob/b5933756b5b488ec51cd56d5984731b6cc347f2b/build-module.py
you can manually trigger by
```
poetry build
```
Depending on the platform, there might be some missing piece that's not allowing the build to succeed.
@JanKrl can you try the above command and paste the output here for debugging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "JanKrl (via GitHub)" <gi...@apache.org>.
JanKrl commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2344182977
> Great to hear. did you clean the env manually or use `make clean`?
I use venv, so I removed `.venv` directory and created it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #1093: Fast Avro Decoder not included in Conda Deployment of pyiceberg
URL: https://github.com/apache/iceberg-python/issues/1093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Fast Avro Decoder not included in Conda Deployment of pyiceberg [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1093:
URL: https://github.com/apache/iceberg-python/issues/1093#issuecomment-2322898924
Im not sure how condo deals with Cython extensions
but here's the relevant code
https://github.com/apache/iceberg-python/blob/e4c1748fee220076f04e35ab2f182dd51ca20705/pyiceberg/avro/decoder.py#L185
https://github.com/apache/iceberg-python/blob/e4c1748fee220076f04e35ab2f182dd51ca20705/build-module.py#L44-L52
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org