You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/01/14 13:02:32 UTC
Re: [PR] Minor: Document `parquet_metadata` function [arrow-datafusion]
alamb commented on code in PR #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852#discussion_r1451729537
##########
docs/source/user-guide/cli.md:
##########
@@ -467,6 +474,66 @@ Available commands inside DataFusion CLI are:
> SET datafusion.execution.batch_size to 1024;
```
+- `parquet_metadata` table function
+
+The `parquet_metadata` table function can be used to inspect detailed metadata
+about a parquet file such as statistics, sizes, and other information. This can
+be helpful to understand how parquet files are structured.
+
+For example, to see information about the `"WatchID"` column in the
+`hits.parquet` file, you can use:
+
+```sql
+SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size
+FROM parquet_metadata('hits.parquet')
+WHERE path_in_schema = '"WatchID"'
+LIMIT 3;
+
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| "WatchID" | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 |
+| "WatchID" | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 |
+| "WatchID" | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+3 rows in set. Query took 0.053 seconds.
+```
+
+The returned table has the following columns for each row for each column chunk
+in the file. Please refer to the [Parquet Documentation] for more information.
+
+[parquet documentation]: https://parquet.apache.org/
+
+| column_name | data_type | Description |
+| ----------------------- | --------- | --------------------------------------------------------------------------------------------------- |
+| filename | Utf8 | Name of the file |
+| row_group_id | Int64 | Row group index the column chunk belongs to |
+| row_group_num_rows | Int64 | Count of rows stored in the row group |
+| row_group_num_columns | Int64 | Total number of columns in the row group (same for all row groups) |
+| row_group_bytes | Int64 | Number of bytes used to store the row group (not including metadata) |
+| column_id | Int64 | ID of the column |
+| file_offset | Int64 | Offset within the file that this column chunk's data begins |
+| num_values | Int64 | Total number of values in this column chunk |
+| path_in_schema | Utf8 | "Path" (column name) of the column chunk in the schema |
+| type | Utf8 | Parquet data type of the column chunk |
+| stats_min | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
+| stats_max | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
+| stats_null_count | Int64 | Number of null values in this column chunk, if stored in the statistics |
+| stats_distinct_count | Int64 | Number of distinct values in this column chunk, if stored in the statistics |
+| stats_min_value | Utf8 | Same as `stats_min` |
Review Comment:
I don't honestly know why the seemingly duplicated columns are present. It was done initially to mirror duckdb which has them. Maybe we should investigate the reason why 🤔
```sql
D create table foo as select * from parquet_metadata('./benchmarks/data/hits.parquet');
D describe table foo;
┌─────────────────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │ null │ key │ default │ extra │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │
├─────────────────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ file_name │ VARCHAR │ YES │ │ │ │
│ row_group_id │ BIGINT │ YES │ │ │ │
│ row_group_num_rows │ BIGINT │ YES │ │ │ │
│ row_group_num_columns │ BIGINT │ YES │ │ │ │
│ row_group_bytes │ BIGINT │ YES │ │ │ │
│ column_id │ BIGINT │ YES │ │ │ │
│ file_offset │ BIGINT │ YES │ │ │ │
│ num_values │ BIGINT │ YES │ │ │ │
│ path_in_schema │ VARCHAR │ YES │ │ │ │
│ type │ VARCHAR │ YES │ │ │ │
│ stats_min │ VARCHAR │ YES │ │ │ │
│ stats_max │ VARCHAR │ YES │ │ │ │
│ stats_null_count │ BIGINT │ YES │ │ │ │
│ stats_distinct_count │ BIGINT │ YES │ │ │ │
│ stats_min_value │ VARCHAR │ YES │ │ │ │
│ stats_max_value │ VARCHAR │ YES │ │ │ │
│ compression │ VARCHAR │ YES │ │ │ │
│ encodings │ VARCHAR │ YES │ │ │ │
│ index_page_offset │ BIGINT │ YES │ │ │ │
│ dictionary_page_offset │ BIGINT │ YES │ │ │ │
│ data_page_offset │ BIGINT │ YES │ │ │ │
│ total_compressed_size │ BIGINT │ YES │ │ │ │
│ total_uncompressed_size │ BIGINT │ YES │ │ │ │
├─────────────────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┤
│ 23 rows 6 columns │
└───────────────────────────────────────────────────────────────────────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org