You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "rotem-ad (via GitHub)" <gi...@apache.org> on 2025/01/01 10:43:51 UTC
[I] Some column statistics are missing after writing data to a table [iceberg-python]
rotem-ad opened a new issue, #1482:
URL: https://github.com/apache/iceberg-python/issues/1482
### Apache Iceberg version
0.8.1 (latest release)
### Please describe the bug 🐞
Following this [Slack thread](https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1734704077106409):
Seems like column statistics are not fully collected when writing data either by using the Arrow Dataframe API or the `add_files` method.
**Example 1:** Stats after using `add_files` method (shown using Trino `show stats`):
```sql
show stats for iceberg_test.tests.my_table_add_files;
+-------------------------+---------+---------------------+--------------+---------+--------------------+-------------------+
|column_name |data_size|distinct_values_count|nulls_fraction|row_count|low_value |high_value |
+-------------------------+---------+---------------------+--------------+---------+--------------------+-------------------+
|id |null |null |0 |null |-9223372031483295744|9223372014179617792|
|location |null |null |null |null |null |null |
|probability |null |null |0 |null |0.2500010132789612 |1.0 |
|null |null |null |null |897861060|null |null |
+-------------------------+---------+---------------------+--------------+---------+--------------------+-------------------+
```
As you can see, `location` column is missing statistics when using `add_files`.
In addition, while calling `add_files` I encountered this error many times (but the operation succeeded):
```
PyArrow statistics missing for column 1 when writing file
```
**Example 2:** Stats after using Arrow Dataframe API to load **the same Parquet files** (shown using Trino `show stats`):
```sql
show stats for iceberg_test.tests.my_table_df;
+-------------------------+-----------+---------------------+--------------+---------+--------------------+-------------------+
|column_name |data_size |distinct_values_count|nulls_fraction|row_count|low_value |high_value |
+-------------------------+-----------+---------------------+--------------+---------+--------------------+-------------------+
|id |null |null |0 |null |-9223372031483295744|9223372014179617792|
|location |14001826255|null |0 |null |null |null |
|probability |null |null |0 |null |0.2500010132789612 |1.0 |
|null |null |null |null |897861060|null |null |
+-------------------------+-----------+---------------------+--------------+---------+--------------------+-------------------+
```
On the other hand, after collecting table statistics using Trino, the column statistics look more complete:
```sql
analyze iceberg_test.tests.my_table_df;
show stats for iceberg_test.tests.my_table_df;
+-------------------------+-----------+---------------------+--------------+---------+--------------------+-------------------+
|column_name |data_size |distinct_values_count|nulls_fraction|row_count|low_value |high_value |
+-------------------------+-----------+---------------------+--------------+---------+--------------------+-------------------+
|id |null |446189731 |0 |null |-9223372031483295744|9223372014179617792|
|location |14001826255|15973993 |0 |null |null |null |
|probability |null |4159975 |0 |null |0.2500010132789612 |1.0 |
|null |null |null |null |897861060|null |null |
+-------------------------+-----------+---------------------+--------------+---------+--------------------+-------------------+
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [X] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Some column statistics are missing after writing data to a table [iceberg-python]
Posted by "amitgilad3 (via GitHub)" <gi...@apache.org>.
amitgilad3 commented on issue #1482:
URL: https://github.com/apache/iceberg-python/issues/1482#issuecomment-2640113224
Hey @rotem-ad , do you have a file we can use to reproduce the issue, i can take a look into this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Some column statistics are missing after writing data to a table [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1482:
URL: https://github.com/apache/iceberg-python/issues/1482#issuecomment-3157017878
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Some column statistics are missing after writing data to a table [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1482:
URL: https://github.com/apache/iceberg-python/issues/1482#issuecomment-2568418644
Thanks for reporting this issue! Both write and add_file uses [`data_file_statistics_from_parquet_metadata`](https://github.com/apache/iceberg-python/blob/5da1f4d6b66cdc689e561d6291abbb757ffa561a/pyiceberg/io/pyarrow.py#L2134-L2138) to generate stats
Do you have the data file so i can try and reproduce this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Some column statistics are missing after writing data to a table [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1482:
URL: https://github.com/apache/iceberg-python/issues/1482#issuecomment-2568422521
Difference between table 1 (add_files) and table 2 (write)
* `data_size` for `location`: `null` vs `14001826255`
* `nulls_fraction` for `location`: `null` vs `0`
Difference between table 2 (write) and table 3 (trino stat collection)
* `distinct_values_count` for `id`, `location`, and `probability`: `null` vs values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Some column statistics are missing after writing data to a table [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #1482:
URL: https://github.com/apache/iceberg-python/issues/1482#issuecomment-3614832848
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org