You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "NGA-TRAN (via GitHub)" <gi...@apache.org> on 2024/05/21 21:21:19 UTC
[I] Incorrect statistics read for struct array in parquet [datafusion]
NGA-TRAN opened a new issue, #10609:
URL: https://github.com/apache/datafusion/issues/10609
### Describe the bug
I found this while adding tests https://github.com/apache/datafusion/pull/10608. The statistics of struct array returns nothing
### To Reproduce
See `test_struct` in https://github.com/apache/datafusion/pull/10608
### Expected behavior
Return some values for the statistics
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect statistics read for struct array in parquet [datafusion]
Posted by "matthewmturner (via GitHub) via github" <gi...@datafusion.apache.org>.
matthewmturner commented on issue #10609:
URL: https://github.com/apache/datafusion/issues/10609#issuecomment-6645904389
actually i see there is active work on this already [here](https://github.com/apache/datafusion/pull/26282) - so im going to drop this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect statistics read for struct array in parquet [datafusion]
Posted by "xinlifoobar (via GitHub)" <gi...@apache.org>.
xinlifoobar commented on issue #10609:
URL: https://github.com/apache/datafusion/issues/10609#issuecomment-2136586252
#8334 Related. The current statistics for structs returns null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect statistics read for struct array in parquet [datafusion]
Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on issue #10609:
URL: https://github.com/apache/datafusion/issues/10609#issuecomment-2125961249
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect statistics read for struct array in parquet [datafusion]
Posted by "Lordworms (via GitHub)" <gi...@apache.org>.
Lordworms commented on issue #10609:
URL: https://github.com/apache/datafusion/issues/10609#issuecomment-2140199393
The problem here is how to effectively deal with nested struct, I don't actually know whether all the columns related to one struct are totally stored in one rowgroup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect statistics read for struct array in parquet [datafusion]
Posted by "matthewmturner (via GitHub) via github" <gi...@datafusion.apache.org>.
matthewmturner commented on issue #10609:
URL: https://github.com/apache/datafusion/issues/10609#issuecomment-6644526907
Im going to pick this up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect statistics read for struct array in parquet [datafusion]
Posted by "matthewmturner (via GitHub) via github" <gi...@datafusion.apache.org>.
matthewmturner commented on issue #10609:
URL: https://github.com/apache/datafusion/issues/10609#issuecomment-6645044066
So a quick initial check and I am seeing some statistics at least for a baseline simple case
```
DataFusion CLI v55.1.0
> COPY (
SELECT
i AS id,
named_struct('name', 'widget' || i, 'qty', i * 10) AS item
FROM generate_series(1, 3) AS t(i)
) TO 'out.parquet' STORED AS PARQUET;
+-------+
| count |
+-------+
| 3 |
+-------+
1 row(s) fetched.
Elapsed 0.005 seconds.
> SELECT * FROM 'out.parquet';
+----+--------------------------+
| id | item |
+----+--------------------------+
| 1 | {name: widget1, qty: 10} |
| 2 | {name: widget2, qty: 20} |
| 3 | {name: widget3, qty: 30} |
+----+--------------------------+
3 row(s) fetched.
Elapsed 0.002 seconds.
> SELECT * FROM parquet_metadata('out.parquet');
+-------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+------------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
| filename | row_group_id | row_group_num_rows | row_group_num_columns | row_group_bytes | column_id | file_offset | num_values | path_in_schema | type | stats_min | stats_max | stats_null_count | stats_distinct_count | stats_min_value | stats_max_value | compression | encodings | index_page_offset | dictionary_page_offset | data_page_offset | total_compressed_size | total_uncompressed_size |
+-------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+------------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
| out.parquet | 0 | 3 | 3 | 198 | 0 | 0 | 3 | "id" | INT64 | 1 | 3 | 0 | NULL | 1 | 3 | ZSTD(ZstdLevel(1)) | [PLAIN, RLE, RLE_DICTIONARY] | NULL | 4 | 47 | 73 | 59 |
| out.parquet | 0 | 3 | 3 | 198 | 1 | 0 | 3 | "item.name" | BYTE_ARRAY | widget1 | widget3 | 0 | NULL | widget1 | widget3 | ZSTD(ZstdLevel(1)) | [PLAIN, RLE, RLE_DICTIONARY] | NULL | 77 | 121 | 80 | 74 |
| out.parquet | 0 | 3 | 3 | 198 | 2 | 0 | 3 | "item.qty" | INT64 | 10 | 30 | 0 | NULL | 10 | 30 | ZSTD(ZstdLevel(1)) | [PLAIN, RLE, RLE_DICTIONARY] | NULL | 157 | 200 | 79 | 65 |
+-------------+--------------+--------------------+-----------------------+-----------------+-----------+-------------+------------+----------------+------------+-----------+-----------+------------------+----------------------+-----------------+-----------------+--------------------+------------------------------+-------------------+------------------------+------------------+-----------------------+-------------------------+
3 row(s) fetched.
Elapsed 0.001 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Incorrect statistics read for struct array in parquet [datafusion]
Posted by "matthewmturner (via GitHub) via github" <gi...@datafusion.apache.org>.
matthewmturner commented on issue #10609:
URL: https://github.com/apache/datafusion/issues/10609#issuecomment-6645189091
Okay but I see missing statistics here
```
> SET datafusion.explain.show_statistics = true;
0 row(s) fetched.
Elapsed 0.000 seconds.
> EXPLAIN ANALYZE SELECT * FROM 'out.parquet';
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan
|
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | DataSourceExec: file_groups={1 group: [[home/matth/OpenSource/datafusion/out.parquet]]}, projection=[id, item], output_ordering=[id@0 ASC NULLS LAST], file_type=parquet, metrics=[output_rows=3, elapsed_compute=226.92µs, output_bytes=96.0 B, output_batches=1, files_ranges_pruned_statistics=1 total → 1 matched, row_groups_pruned_statistics=1 total → 1 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, bytes_processed=1228.0 B, bytes_scanned=232.0 B, file_open_errors=0, file_scan_errors=0, files_opened=1, files_processed=1, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_pruned_dynamic_filter=0, predicate_cache_inner_records=0, predicate_cache_records=0, bloom_filter_eval_time=91ns, metadata_load_time=50.63µs, page_index_eval_time=2ns, row_pushdown_eval_time=2ns, stati
stics_eval_time=2ns, time_elapsed_opening=167.93µs, time_elapsed_processing=475.46µs, time_elapsed_scanning_total=419.28µs, time_elapsed_scanning_until_data=368.95µs, output_rows_skew=0%, scan_efficiency_ratio=18.89% (232/1.23 K)], statistics=[Rows=Exact(3), Bytes=Absent, [(Col[0]: Min=Exact(Int64(1)) Max=Exact(Int64(3)) Null=Exact(0) ScanBytes=Exact(24)),(Col[1]: Min=Inexact(Struct({name:,qty:})) Max=Inexact(Struct({name:,qty:})))]] |
| |
|
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.003 seconds.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org