You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bigluck (via GitHub)" <gi...@apache.org> on 2023/03/03 08:38:49 UTC
[GitHub] [iceberg] bigluck commented on issue #6505: Python: Infer Iceberg schema from the Parquet file
bigluck commented on issue #6505:
URL: https://github.com/apache/iceberg/issues/6505#issuecomment-1453166224
Ciao @Fokko, maybe I'm facing a similar issue, but I'm a bit confused.
I'm using Glue and I'm querying an iceberg table created with Dremio.
The table in question derives from the open dataset of NY taxis.
```python
import os
from pyiceberg.catalog import load_glue
catalog = load_glue(name='biglake', conf={})
table = catalog.load_table('biglake.taxi_dremio_by_month')
print(table.identifier)
print(table.metadata)
print(table.metadata_location)
con = table.scan().to_duckdb(table_name='taxi')
print(con.execute('SELECT COUNT(*) FROM taxi').fetchall())
```
This is the output:
```
('biglake', 'taxi_dremio_by_month')
location='s3://my-s3-bucket/biglake/taxi_dremio_by_month'
table_uuid=UUID('80a4d129-4919-4b2a-8784-ec845a853130')
last_updated_ms=1677495898549
last_column_id=24
schemas=[
Schema(
NestedField(field_id=1, name='hvfhs_license_num', field_type=StringType(), required=False),
NestedField(field_id=2, name='dispatching_base_num', field_type=StringType(), required=False),
NestedField(field_id=3, name='originating_base_num', field_type=StringType(), required=False),
NestedField(field_id=4, name='request_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=5, name='on_scene_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=6, name='pickup_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=7, name='dropoff_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=8, name='PULocationID', field_type=LongType(), required=False),
NestedField(field_id=9, name='DOLocationID', field_type=LongType(), required=False),
NestedField(field_id=10, name='trip_miles', field_type=DoubleType(), required=False),
NestedField(field_id=11, name='trip_time', field_type=LongType(), required=False),
NestedField(field_id=12, name='base_passenger_fare', field_type=DoubleType(), required=False),
NestedField(field_id=13, name='tolls', field_type=DoubleType(), required=False),
NestedField(field_id=14, name='bcf', field_type=DoubleType(), required=False),
NestedField(field_id=15, name='sales_tax', field_type=DoubleType(), required=False),
NestedField(field_id=16, name='congestion_surcharge', field_type=DoubleType(), required=False),
NestedField(field_id=17, name='airport_fee', field_type=IntegerType(), required=False),
NestedField(field_id=18, name='tips', field_type=DoubleType(), required=False),
NestedField(field_id=19, name='driver_pay', field_type=DoubleType(), required=False),
NestedField(field_id=20, name='shared_request_flag', field_type=StringType(), required=False),
NestedField(field_id=21, name='shared_match_flag', field_type=StringType(), required=False),
NestedField(field_id=22, name='access_a_ride_flag', field_type=StringType(), required=False),
NestedField(field_id=23, name='wav_request_flag', field_type=StringType(), required=False),
NestedField(field_id=24, name='wav_match_flag', field_type=StringType(), required=False),
schema_id=0,
identifier_field_ids=[]
)
]
current_schema_id=0
partition_specs=[
PartitionSpec(
PartitionField(source_id=4, field_id=1000, transform=MonthTransform(), name='request_datetime_month'),
spec_id=0
)
]
default_spec_id=0
last_partition_id=1000
properties={
'compatibility.snapshot-id-inheritance.enabled': 'true',
'commit.manifest.target-size-bytes': '153600'
}
current_snapshot_id=666682962113515828
snapshots=[
Snapshot(
snapshot_id=666682962113515828,
parent_snapshot_id=None,
sequence_number=None,
timestamp_ms=1677495898549,
manifest_list='s3://my-s3-bucket/biglake/taxi_dremio_by_month/metadata/snap-666682962113515828-1-133eb191-5a2e-43da-a773-9f87eeb6b495.avro',
summary=Summary(
Operation.APPEND, **{
'added-data-files': '260',
'added-records': '745287023',
'total-records': '745287023',
'total-files-size': '0',
'total-data-files': '260',
'total-delete-files': '0',
'total-position-deletes': '0',
'total-equality-deletes': '0'
}
),
schema_id=0
)
]
snapshot_log=[
SnapshotLogEntry(
snapshot_id='666682962113515828',
timestamp_ms=1677495898549
)
]
metadata_log=[]
sort_orders=[SortOrder(order_id=0)]
default_sort_order_id=0
refs={
'main': SnapshotRef(
snapshot_id=666682962113515828,
snapshot_ref_type=SnapshotRefType.BRANCH,
min_snapshots_to_keep=None,
max_snapshot_age_ms=None,
max_ref_age_ms=None
)
}
format_version=1
schema_=Schema(
NestedField(field_id=1, name='hvfhs_license_num', field_type=StringType(), required=False),
NestedField(field_id=2, name='dispatching_base_num', field_type=StringType(), required=False),
NestedField(field_id=3, name='originating_base_num', field_type=StringType(), required=False),
NestedField(field_id=4, name='request_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=5, name='on_scene_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=6, name='pickup_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=7, name='dropoff_datetime', field_type=TimestamptzType(), required=False),
NestedField(field_id=8, name='PULocationID', field_type=LongType(), required=False),
NestedField(field_id=9, name='DOLocationID', field_type=LongType(), required=False),
NestedField(field_id=10, name='trip_miles', field_type=DoubleType(), required=False),
NestedField(field_id=11, name='trip_time', field_type=LongType(), required=False),
NestedField(field_id=12, name='base_passenger_fare', field_type=DoubleType(), required=False),
NestedField(field_id=13, name='tolls', field_type=DoubleType(), required=False),
NestedField(field_id=14, name='bcf', field_type=DoubleType(), required=False),
NestedField(field_id=15, name='sales_tax', field_type=DoubleType(), required=False),
NestedField(field_id=16, name='congestion_surcharge', field_type=DoubleType(), required=False),
NestedField(field_id=17, name='airport_fee', field_type=IntegerType(), required=False),
NestedField(field_id=18, name='tips', field_type=DoubleType(), required=False),
NestedField(field_id=19, name='driver_pay', field_type=DoubleType(), required=False),
NestedField(field_id=20, name='shared_request_flag', field_type=StringType(), required=False),
NestedField(field_id=21, name='shared_match_flag', field_type=StringType(), required=False),
NestedField(field_id=22, name='access_a_ride_flag', field_type=StringType(), required=False),
NestedField(field_id=23, name='wav_request_flag', field_type=StringType(), required=False),
NestedField(field_id=24, name='wav_match_flag', field_type=StringType(), required=False),
schema_id=0,
identifier_field_ids=[]
)
partition_spec=[
{
'name': 'request_datetime_month',
'transform': 'month',
'source-id': 4,
'field-id': 1000
}
]
s3://my-s3-bucket/biglake/taxi_dremio_by_month/metadata/00000-2cf6fb41-1c37-4e7c-a35b-3ab1c4670b45.metadata.json
```
And then it crashes:
```
Traceback (most recent call last):
File "/home/ubuntu/src/query.py", line 17, in <module>
con = table.scan().to_duckdb(table_name='taxi')
File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/table/__init__.py", line 360, in to_duckdb
con.register(table_name, self.to_arrow())
File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/table/__init__.py", line 349, in to_arrow
return project_table(
File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py", line 552, in project_table
tables = pool.starmap(
File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py", line 491, in _file_to_table
raise ValueError(
ValueError: Iceberg schema is not embedded into the Parquet file, see https://github.com/apache/iceberg/issues/6505
```
I'm confused because the query is a simple `COUNT(*)`, and I thought pyiceber would read the metadata stored on the metadata folder to get the number of records.
<img width="1336" alt="Screenshot 2023-03-03 at 09 31 39" src="https://user-images.githubusercontent.com/1511095/222671001-9cbdd901-3a7e-4b80-a8a5-385aa1f34653.png">
I've also tested PR #6997, but the python operator crashed:
```
s3://my-s3-bucket/biglake/taxi_dremio_by_month/metadata/00000-2cf6fb41-1c37-4e7c-a35b-3ab1c4670b45.metadata.json
Killed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org