You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/12/15 14:19:36 UTC
[GitHub] [iceberg] joshuarobinson opened a new issue, #6434: PyIceberg support for UUID types
joshuarobinson opened a new issue, #6434:
URL: https://github.com/apache/iceberg/issues/6434
### Feature Request / Improvement
Currently, pyiceberg 0.2.0 fails on creating a table scan for any table (that I have at least) with UUID columns.
The root problem seems to be (thanks @Fokko for explanation) that UUID is encoded as a string in avro but as a fixed-width type in Iceberg.
The failure looks like this:
```
ValueError: Unknown logical/physical type combination: {'type': 'fixed', 'name': 'uuid_fixed', 'size': 16, 'logicalType': 'uuid'}
```
The full stacktrace:
```
Traceback (most recent call last):
File "/read.py", line 16, in <module>
print(tbl.scan(selected_fields=("path","device")).to_arrow().to_pandas().head())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 350, in to_arrow
for task in self.plan_files():
File "/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 335, in plan_files
yield from (FileScanTask(file) for file in matching_partition_files)
File "/usr/local/lib/python3.11/site-packages/pyiceberg/table/__init__.py", line 335, in <genexpr>
yield from (FileScanTask(file) for file in matching_partition_files)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line 149, in <genexpr>
return (entry.data_file for entry in live_entries(input_file))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line 145, in <genexpr>
return (entry for entry in read_manifest_entry(input_file) if entry.status != ManifestEntryStatus.DELETED)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/manifest.py", line 137, in read_manifest_entry
with AvroFile(input_file) as reader:
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py", line 136, in __enter__
self.schema = self.header.get_schema()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/avro/file.py", line 85, in get_schema
return AvroSchemaConversion().avro_to_iceberg(avro_schema)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 119, in avro_to_iceberg
return Schema(*[self._convert_field(field) for field in avro_schema["fields"]], schema_id=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 119, in <listcomp>
return Schema(*[self._convert_field(field) for field in avro_schema["fields"]], schema_id=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 227, in _convert_field
field_type=self._convert_schema(plain_type),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 196, in _convert_schema
return self._convert_record_type(avro_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 284, in _convert_record_type
return StructType(*[self._convert_field(field) for field in record_type["fields"]])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 284, in <listcomp>
return StructType(*[self._convert_field(field) for field in record_type["fields"]])
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 227, in _convert_field
field_type=self._convert_schema(plain_type),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 196, in _convert_schema
return self._convert_record_type(avro_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 284, in _convert_record_type
return StructType(*[self._convert_field(field) for field in record_type["fields"]])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 284, in <listcomp>
return StructType(*[self._convert_field(field) for field in record_type["fields"]])
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 227, in _convert_field
field_type=self._convert_schema(plain_type),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 189, in _convert_schema
return self._convert_logical_type(avro_type)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyiceberg/utils/schema_conversion.py", line 368, in _convert_logical_type
raise ValueError(f"Unknown logical/physical type combination: {avro_logical_type}")
ValueError: Unknown logical/physical type combination: {'type': 'fixed', 'name': 'uuid_fixed', 'size': 16, 'logicalType': 'uuid'}
```
The problem appears even if I'm not selecting the UUID columns as part of the scan.
### Query engine
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue closed issue #6434: PyIceberg support for UUID types
Posted by GitBox <gi...@apache.org>.
rdblue closed issue #6434: PyIceberg support for UUID types
URL: https://github.com/apache/iceberg/issues/6434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org