You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "cgbur (via GitHub)" <gi...@apache.org> on 2024/05/07 23:09:52 UTC
[I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
cgbur opened a new issue, #716:
URL: https://github.com/apache/iceberg-python/issues/716
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
When using the `add_files` table api, the parquet metadata needs to be read and a mapping of `Dict[str, int]` is used by [`data_file_statistics_from_parquet_metadata`](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/pyarrow.py#L1670) in order to link the field ID to the name in the parquet file for statistics collection. However during [the mapping lookup](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/pyarrow.py#L1727) I was receiving an error that a key was not present.
My schema contains one of the following (its a subfield of a `Details` struct which is important for the full name later):
```
extras: large_list<item: struct<key: string not null, value: string>> not null
child 0, item: struct<key: string not null, value: string>
child 0, key: string not null
child 1, value: string
```
Which based on the parquet schema path definition has a path of:
```
Details.extras.list.item.key
Details.extras.list.item.value
```
The issue is that the [`parquet_path_to_id_mapping`](https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/pyarrow.py#L1587) returns a mapping for these two fields as follows:
```
Details.extras.list.element.key -> 189
Details.extras.list.element.value -> 190
```
So, the issue appears to be that the visitor for constructing the schema paths is incorrectly using `element` instead of `item` as expected in the parquet schema paths. I am not sure how this manifests yet, as I have not dug into it too closely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "cgbur (via GitHub)" <gi...@apache.org>.
cgbur commented on issue #716:
URL: https://github.com/apache/iceberg-python/issues/716#issuecomment-2101251170
Ah, confusingly there appears to be writer differences that cause the issue. My Rust pyarrow implementation matches when polars has `pyarrow=True`.
```python
import polars as pl
import pyarrow.parquet as pq
df = pl.DataFrame(
{
"a": [[{"a": 1}, {"a": 2}], [{"a": 3}]],
}
)
def print_schema_path(path, col_name):
metadata = pq.read_metadata(path)
for group_number in range(metadata.num_row_groups):
row_group = metadata.row_group(group_number)
for column_number in range(row_group.num_columns):
column = row_group.column(column_number)
if column.path_in_schema.startswith(col_name):
print(f"path_in_schema: {column.path_in_schema}")
df.write_parquet("example.parquet", use_pyarrow=False)
print("with polars")
print(pq.read_schema("example.parquet"))
print_schema_path("example.parquet", "a")
df.write_parquet("example.parquet", use_pyarrow=True)
print("with pyarrow")
print(pq.read_schema("example.parquet"))
print_schema_path("example.parquet", "a")
```
```
with polars
a: large_list<item: struct<a: int64>>
child 0, item: struct<a: int64>
child 0, a: int64
path_in_schema: a.list.item.a
with pyarrow
a: large_list<element: struct<a: int64>>
child 0, element: struct<a: int64>
child 0, a: int64
path_in_schema: a.list.element.a
```
Perhaps the visitor is not respecting the name used in the schema? Or there is a mismatch in the method used to acquire between the iceberg and parquet change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "syun64 (via GitHub)" <gi...@apache.org>.
syun64 commented on issue #716:
URL: https://github.com/apache/iceberg-python/issues/716#issuecomment-2170966077
Hi @cgbur and @felixscherz thank you for raising this and taking this investigation further. I'm not a polars user myself, but the difference in the behavior is quite interesting, and I think there would be value in trying to fix this issue.
I just read the `write_table` API documentation in `pyarrow.parquet` and found something rather interesting:
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table
If you check the documentation for `use_compliant_nested_type` flag, it mentions that having `element` as the single item field name is the parquet compliant format as specified here on the [Parquet Spec for Nested Types](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#nested-types). PyArrow defaults to using this flag and writes the list element field name as `element`.
For some reason it looks like polars has decided to use `item` - the non-parquet compliant list element name instead. While I'm curious about why the polars community has decided to go this route, I also think supporting both `element` or `item` name in the visitor may not be the worst thing, just to increase our scope of support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #716:
URL: https://github.com/apache/iceberg-python/issues/716#issuecomment-2549967377
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #716: `parquet_path_to_id_mapping` generates incorrect path for List types
URL: https://github.com/apache/iceberg-python/issues/716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "felixscherz (via GitHub)" <gi...@apache.org>.
felixscherz commented on issue #716:
URL: https://github.com/apache/iceberg-python/issues/716#issuecomment-2118806030
Hi, I investigated this a bit further and it seems to be related to the way the visitor works as @cgbur suggested. Here is what I tried:
```python
def test_parquet_path_to_id_mapping():
# set field name to "item"
pyarrow_list = pa.schema([
pa.field("extras", pa.list_(pa.field("item", pa.struct([pa.field("key", pa.string()), pa.field("value", pa.string())]))))
])
# this is called during table creation
schema = Catalog._convert_schema_if_needed(pyarrow_list)
mapping = parquet_path_to_id_mapping(schema)
assert "extras.list.item.key" in mapping
```
The mapping that `Catalog._convert_schema_if_needed` creates looks like this:
```python
{'extras.list.element.key': -1, 'extras.list.element.value': -1}
```
Looking into the visitor I found the method dealing with list types sets a default field name of "elements".
https://github.com/apache/iceberg-python/blob/20c273104257f0a1ccd74a09f6d4601643115ffd/pyiceberg/io/pyarrow.py#L865-L870
https://github.com/apache/iceberg-python/blob/20c273104257f0a1ccd74a09f6d4601643115ffd/pyiceberg/io/pyarrow.py#L172
So we lose the information on the field name of the value field, setting it to "elements".
Unfortunately I haven't found a way to access the field name as both [pyarrow.lib.ListType](https://arrow.apache.org/docs/python/generated/pyarrow.ListType.html) and [pyarrow.lib.DataType](https://arrow.apache.org/docs/python/generated/pyarrow.DataType.html#pyarrow.DataType) don't seem to make that available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "cgbur (via GitHub)" <gi...@apache.org>.
cgbur commented on issue #716:
URL: https://github.com/apache/iceberg-python/issues/716#issuecomment-2101184351
Here is a complete example recreating the error. Here I am using polars to make the table which results in the same schema that I am producing with pyarrow.
```python
import polars as pl
from pyiceberg.catalog.sql import SqlCatalog
import pyarrow.parquet as pq
import os
import shutil
pl.DataFrame(
{
"a": [[{"a": 1}, {"a": 2}], [{"a": 3}]],
}
).write_parquet("example.parquet")
warehouse_path = "/tmp/warehouse"
# wipe the warehouse
if os.path.exists(warehouse_path):
shutil.rmtree(warehouse_path)
os.makedirs(warehouse_path)
catalog = SqlCatalog(
"default",
**{
"uri": f"sqlite:///{warehouse_path}/pyiceberg_catalog.db",
"warehouse": f"file://{warehouse_path}",
},
)
df = pq.read_table("example.parquet")
catalog.create_namespace("default")
table = catalog.create_table(
"default.webserver",
schema=df.schema,
)
table.add_files(["example.parquet"])
```
And here is the error. The top two lines were debug statements showing how the mapping file has the incorrect path.
```python
print(f"column mappings {len(parquet_column_mapping)}")
print(parquet_column_mapping)
```
```
column mappings 1
{'a.list.element.a': 3}
Traceback (most recent call last):
File "/home/cgbur/pyice-test/failure.py", line 36, in <module>
table.add_files(["example.parquet"])
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/table/__init__.py", line 1355, in add_files
tx.add_files(file_paths=file_paths, snapshot_properties=snapshot_properties)
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/table/__init__.py", line 462, in add_files
for data_file in data_files:
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/table/__init__.py", line 2737, in _parquet_files_to_data_files
yield from parquet_files_to_data_files(io=io, table_metadata=table_metadata, file_paths=iter(file_paths))
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/io/pyarrow.py", line 1869, in parquet_files_to_data_files
statistics = data_file_statistics_from_parquet_metadata(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/io/pyarrow.py", line 1734, in data_file_statistics_from_parquet_metadata
field_id = parquet_column_mapping[column.path_in_schema]
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'a.list.item.a'
```
You can see how the parquet `path_in_schema` using the `item` instead of `element`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #716:
URL: https://github.com/apache/iceberg-python/issues/716#issuecomment-2100409045
@cgbur Thanks for raising this. Could you share the stack trace that you're seeing?
I tried to reproduce it, but it works on my end:
```python
def test_data_file_statistics_from_parquet_metadata_list(tmp_path_factory: pytest.TempPathFactory) -> None:
pyarrow_list = pa.schema([
pa.field("extras", pa.list_(pa.field("element", pa.struct([pa.field("key", pa.string()), pa.field("value", pa.string())]))))
])
tbl = pa.Table.from_pylist([{'some_list': [{"key": "a", "value": "b"}]}], schema=pyarrow_list)
file_path = tmp_path_factory.mktemp('test_statistics') / "test.parquet"
pq.write_table(tbl, file_path)
parquet_metadata = pq.read_metadata(file_path)
iceberg_schema = Schema(
NestedField(
1,
"extras",
ListType(
10,
StructType(
NestedField(10, "key", StringType()),
NestedField(11, "value", StringType()),
),
element_required=False,
),
)
)
statistics = data_file_statistics_from_parquet_metadata(
parquet_metadata=parquet_metadata,
stats_columns=compute_statistics_plan(iceberg_schema, EMPTY_DICT),
parquet_column_mapping=parquet_path_to_id_mapping(iceberg_schema),
)
assert statistics == DataFileStatistics(
record_count=1,
column_sizes={10: 51, 11: 51},
value_counts={10: 1, 11: 1},
null_value_counts={10: 1, 11: 1},
nan_value_counts={},
column_aggregates={},
split_offsets=[4],
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] `parquet_path_to_id_mapping` generates incorrect path for List types [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #716:
URL: https://github.com/apache/iceberg-python/issues/716#issuecomment-2566767375
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org