You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ZENOTME (via GitHub)" <gi...@apache.org> on 2023/12/27 08:17:22 UTC
[I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]
ZENOTME opened a new issue, #241:
URL: https://github.com/apache/iceberg-python/issues/241
### Question
[Icelake](https://github.com/icelake-io/icelake) is a rust sdk and it use avro-rs. I find that it will report `EOF: read 1 bytes` (caused by: https://github.com/apache/iceberg-python/blob/088ee4076e733422d7d50491cab8860ae08426af/pyiceberg/avro/decoder_fast.pyx#L110) when load a table write by icelake. But the table can be load using iceberg-java.
here is the error log:
```
Using python3 (3.11.5)
Traceback (most recent call last):
File "/Users/ze/Project/icelake/testdata/python/check_pyiceberg.py", line 27, in <module>
con1 = t1.scan().to_duckdb(table_name="t1")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 904, in to_duckdb
con.register(table_name, self.to_arrow())
^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 889, in to_arrow
self.plan_files(),
^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 849, in plan_files
for manifest_entry in chain(
^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 851, in <lambda>
lambda args: _open_manifest(*args),
^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 720, in _open_manifest
for manifest_entry in manifest.fetch_manifest_entry(io, discard_deleted=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/manifest.py", line 354, in fetch_manifest_entry
return [
^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/manifest.py", line 354, in <listcomp>
return [
^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/file.py", line 215, in __next__
return self.__next__()
^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/file.py", line 207, in __next__
return next(self.block)
^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/file.py", line 122, in __next__
return self.reader.read(self.block_decoder)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/reader.py", line 334, in read
struct[pos] = field_reader(decoder) # later: pass reuse in here
^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/reader.py", line 334, in read
struct[pos] = field_reader(decoder) # later: pass reuse in here
^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/reader.py", line 281, in read
return self.option.read(decoder)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/reader.py", line 459, in read
return self._read_int_int(decoder)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ze/Library/Caches/pypoetry/virtualenvs/icelake-integration-tests-FuW_3NoK-py3.11/lib/python3.11/site-packages/pyiceberg/avro/reader.py", line 450, in _read_int_int
block_count = decoder.read_int()
^^^^^^^^^^^^^^^^^^
File "pyiceberg/avro/decoder_fast.pyx", line 86, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_int
File "pyiceberg/avro/decoder_fast.pyx", line 93, in pyiceberg.avro.decoder_fast.CythonBinaryDecoder.read_int
EOFError: EOF: read 1 bytes
```
I'm not sure what's wrong here. May be there is bug in icelake. But I want to make sure that there is no bug in pyiceberg in this case. Seems because in this read_int, the buffer is empty so it throw a exception, Is possible to treat the count as 0 when the buffer is empty here?🤔
https://github.com/apache/iceberg-python/blob/088ee4076e733422d7d50491cab8860ae08426af/pyiceberg/avro/reader.py#L450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #241:
URL: https://github.com/apache/iceberg-python/issues/241#issuecomment-1870247267
> @ZENOTME Thanks for raising this issue! Can you share an example Avro file that has the bug?
Sure, Thanks for your help!
[b404ad7a-4de1-409c-be71-8c799a88e98b-m1.avro.zip](https://github.com/apache/iceberg-python/files/13778381/b404ad7a-4de1-409c-be71-8c799a88e98b-m1.avro.zip) here is the manifest file. I guess is the file may have bug.
[t1.zip](https://github.com/apache/iceberg-python/files/13778409/t1.zip)here is the whole table.
And following is the code:
```
catalog = load_catalog(
# "local",
**{
"type": "rest",
"uri": args.resturl,
"s3.endpoint": args.s3url,
"s3.access-key-id": "admin",
"s3.secret-access-key": "password",
},
)
t1 = catalog.load_table(f"s1.{args.t1}")
con1 = t1.scan().to_duckdb(table_name="t1")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #241:
URL: https://github.com/apache/iceberg-python/issues/241#issuecomment-1870117506
@ZENOTME Thanks for raising this issue! Can you share an example Avro file that has the bug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #241:
URL: https://github.com/apache/iceberg-python/issues/241#issuecomment-1876600551
Ah that makes sense, with the bytes it will first read the length of the bytes to be read. I didn't get to the debugging yet since it is quite tedious :D Great find!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #241:
URL: https://github.com/apache/iceberg-python/issues/241#issuecomment-1876278634
After some investigation, I realized that this is caused by a bug in icelake, which converts to bytes instead of fixed when converting to avro schema. I'll close this for now, feel free to open it when necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] EOF: read 1 bytes when load manifest write by icelake [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #241: EOF: read 1 bytes when load manifest write by icelake
URL: https://github.com/apache/iceberg-python/issues/241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org