You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "bigluck (via GitHub)" <gi...@apache.org> on 2023/03/03 08:38:49 UTC

[GitHub] [iceberg] bigluck commented on issue #6505: Python: Infer Iceberg schema from the Parquet file

bigluck commented on issue #6505:
URL: https://github.com/apache/iceberg/issues/6505#issuecomment-1453166224

   Ciao @Fokko, maybe I'm facing a similar issue, but I'm a bit confused.
   I'm using Glue and I'm querying an iceberg table created with Dremio.
   
   The table in question derives from the open dataset of NY taxis.
   
   ```python
   import os
   from pyiceberg.catalog import load_glue
   
   catalog = load_glue(name='biglake', conf={})
   table = catalog.load_table('biglake.taxi_dremio_by_month')
   
   print(table.identifier)
   print(table.metadata)
   print(table.metadata_location)
   
   con = table.scan().to_duckdb(table_name='taxi')
   print(con.execute('SELECT COUNT(*) FROM taxi').fetchall())
   ```
   
   This is the output:
   
   ```
   ('biglake', 'taxi_dremio_by_month')
   
   location='s3://my-s3-bucket/biglake/taxi_dremio_by_month'
   table_uuid=UUID('80a4d129-4919-4b2a-8784-ec845a853130')
   last_updated_ms=1677495898549
   last_column_id=24
   schemas=[
   	Schema(
   		NestedField(field_id=1, name='hvfhs_license_num', field_type=StringType(), required=False),
   		NestedField(field_id=2, name='dispatching_base_num', field_type=StringType(), required=False),
   		NestedField(field_id=3, name='originating_base_num', field_type=StringType(), required=False),
   		NestedField(field_id=4, name='request_datetime', field_type=TimestamptzType(), required=False),
   		NestedField(field_id=5, name='on_scene_datetime', field_type=TimestamptzType(), required=False),
   		NestedField(field_id=6, name='pickup_datetime', field_type=TimestamptzType(), required=False),
   		NestedField(field_id=7, name='dropoff_datetime', field_type=TimestamptzType(), required=False),
   		NestedField(field_id=8, name='PULocationID', field_type=LongType(), required=False),
   		NestedField(field_id=9, name='DOLocationID', field_type=LongType(), required=False),
   		NestedField(field_id=10, name='trip_miles', field_type=DoubleType(), required=False),
   		NestedField(field_id=11, name='trip_time', field_type=LongType(), required=False),
   		NestedField(field_id=12, name='base_passenger_fare', field_type=DoubleType(), required=False),
   		NestedField(field_id=13, name='tolls', field_type=DoubleType(), required=False),
   		NestedField(field_id=14, name='bcf', field_type=DoubleType(), required=False),
   		NestedField(field_id=15, name='sales_tax', field_type=DoubleType(), required=False),
   		NestedField(field_id=16, name='congestion_surcharge', field_type=DoubleType(), required=False),
   		NestedField(field_id=17, name='airport_fee', field_type=IntegerType(), required=False),
   		NestedField(field_id=18, name='tips', field_type=DoubleType(), required=False),
   		NestedField(field_id=19, name='driver_pay', field_type=DoubleType(), required=False),
   		NestedField(field_id=20, name='shared_request_flag', field_type=StringType(), required=False),
   		NestedField(field_id=21, name='shared_match_flag', field_type=StringType(), required=False),
   		NestedField(field_id=22, name='access_a_ride_flag', field_type=StringType(), required=False),
   		NestedField(field_id=23, name='wav_request_flag', field_type=StringType(), required=False),
   		NestedField(field_id=24, name='wav_match_flag', field_type=StringType(), required=False),
   		schema_id=0,
   		identifier_field_ids=[]
   	)
   ]
   current_schema_id=0
   partition_specs=[
   	PartitionSpec(
   		PartitionField(source_id=4, field_id=1000, transform=MonthTransform(), name='request_datetime_month'),
   		spec_id=0
   	)
   ]
   default_spec_id=0
   last_partition_id=1000
   properties={
   	'compatibility.snapshot-id-inheritance.enabled': 'true',
   	'commit.manifest.target-size-bytes': '153600'
   }
   current_snapshot_id=666682962113515828
   snapshots=[
   	Snapshot(
   		snapshot_id=666682962113515828,
   		parent_snapshot_id=None,
   		sequence_number=None,
   		timestamp_ms=1677495898549,
   		manifest_list='s3://my-s3-bucket/biglake/taxi_dremio_by_month/metadata/snap-666682962113515828-1-133eb191-5a2e-43da-a773-9f87eeb6b495.avro',
   		summary=Summary(
   			Operation.APPEND, **{
   				'added-data-files': '260',
   				'added-records': '745287023',
   				'total-records': '745287023',
   				'total-files-size': '0',
   				'total-data-files': '260',
   				'total-delete-files': '0',
   				'total-position-deletes': '0',
   				'total-equality-deletes': '0'
   			}
   		),
   		schema_id=0
   	)
   ]
   snapshot_log=[
   	SnapshotLogEntry(
   		snapshot_id='666682962113515828',
   		timestamp_ms=1677495898549
   	)
   ]
   metadata_log=[]
   sort_orders=[SortOrder(order_id=0)]
   default_sort_order_id=0
   refs={
   	'main': SnapshotRef(
   		snapshot_id=666682962113515828,
   		snapshot_ref_type=SnapshotRefType.BRANCH,
   		min_snapshots_to_keep=None,
   		max_snapshot_age_ms=None,
   		max_ref_age_ms=None
   	)
   }
   format_version=1
   schema_=Schema(
   	NestedField(field_id=1, name='hvfhs_license_num', field_type=StringType(), required=False),
   	NestedField(field_id=2, name='dispatching_base_num', field_type=StringType(), required=False),
   	NestedField(field_id=3, name='originating_base_num', field_type=StringType(), required=False),
   	NestedField(field_id=4, name='request_datetime', field_type=TimestamptzType(), required=False),
   	NestedField(field_id=5, name='on_scene_datetime', field_type=TimestamptzType(), required=False),
   	NestedField(field_id=6, name='pickup_datetime', field_type=TimestamptzType(), required=False),
   	NestedField(field_id=7, name='dropoff_datetime', field_type=TimestamptzType(), required=False),
   	NestedField(field_id=8, name='PULocationID', field_type=LongType(), required=False),
   	NestedField(field_id=9, name='DOLocationID', field_type=LongType(), required=False),
   	NestedField(field_id=10, name='trip_miles', field_type=DoubleType(), required=False),
   	NestedField(field_id=11, name='trip_time', field_type=LongType(), required=False),
   	NestedField(field_id=12, name='base_passenger_fare', field_type=DoubleType(), required=False),
   	NestedField(field_id=13, name='tolls', field_type=DoubleType(), required=False),
   	NestedField(field_id=14, name='bcf', field_type=DoubleType(), required=False),
   	NestedField(field_id=15, name='sales_tax', field_type=DoubleType(), required=False),
   	NestedField(field_id=16, name='congestion_surcharge', field_type=DoubleType(), required=False),
   	NestedField(field_id=17, name='airport_fee', field_type=IntegerType(), required=False),
   	NestedField(field_id=18, name='tips', field_type=DoubleType(), required=False),
   	NestedField(field_id=19, name='driver_pay', field_type=DoubleType(), required=False),
   	NestedField(field_id=20, name='shared_request_flag', field_type=StringType(), required=False),
   	NestedField(field_id=21, name='shared_match_flag', field_type=StringType(), required=False),
   	NestedField(field_id=22, name='access_a_ride_flag', field_type=StringType(), required=False),
   	NestedField(field_id=23, name='wav_request_flag', field_type=StringType(), required=False),
   	NestedField(field_id=24, name='wav_match_flag', field_type=StringType(), required=False),
   	schema_id=0,
   	identifier_field_ids=[]
   )
   partition_spec=[
   	{
   		'name': 'request_datetime_month',
   		'transform': 'month',
   		'source-id': 4,
   		'field-id': 1000
   	}
   ]
   
   s3://my-s3-bucket/biglake/taxi_dremio_by_month/metadata/00000-2cf6fb41-1c37-4e7c-a35b-3ab1c4670b45.metadata.json
   ```
   
   And then it crashes:
   ```
   Traceback (most recent call last):
     File "/home/ubuntu/src/query.py", line 17, in <module>
       con = table.scan().to_duckdb(table_name='taxi')
     File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/table/__init__.py", line 360, in to_duckdb
       con.register(table_name, self.to_arrow())
     File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/table/__init__.py", line 349, in to_arrow
       return project_table(
     File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py", line 552, in project_table
       tables = pool.starmap(
     File "/usr/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
       return self._map_async(func, iterable, starmapstar, chunksize).get()
     File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
       raise self._value
     File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
       result = (True, func(*args, **kwds))
     File "/usr/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
       return list(itertools.starmap(args[0], args[1]))
     File "/home/ubuntu/src/.venv/lib/python3.10/site-packages/pyiceberg/io/pyarrow.py", line 491, in _file_to_table
       raise ValueError(
   ValueError: Iceberg schema is not embedded into the Parquet file, see https://github.com/apache/iceberg/issues/6505
   ```
   
   I'm confused because the query is a simple `COUNT(*)`, and I thought pyiceber would read the metadata stored on the metadata folder to get the number of records.
   
   <img width="1336" alt="Screenshot 2023-03-03 at 09 31 39" src="https://user-images.githubusercontent.com/1511095/222671001-9cbdd901-3a7e-4b80-a8a5-385aa1f34653.png">
   
   I've also tested PR #6997, but the python operator crashed:
   
   ```
   s3://my-s3-bucket/biglake/taxi_dremio_by_month/metadata/00000-2cf6fb41-1c37-4e7c-a35b-3ab1c4670b45.metadata.json
   Killed
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org