You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/01/13 12:57:42 UTC
[PR] Minor: Document parquet_metadata function [arrow-datafusion]
alamb opened a new pull request, #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852
## Which issue does this PR close?
Part of #7013
## Rationale for this change
I was writing a blog post about DataFusion https://github.com/apache/arrow-site/pull/457 and I wanted to highlight this feature that @Veeupup added in https://github.com/apache/arrow-datafusion/issues/8367 but it wasn't documented
## What changes are included in this PR?
Document the function
## Are these changes tested?
N/A
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
More docs
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Document parquet_metadata function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852#discussion_r1451592588
##########
docs/source/user-guide/cli.md:
##########
@@ -467,6 +474,66 @@ Available commands inside DataFusion CLI are:
> SET datafusion.execution.batch_size to 1024;
```
+- `parquet_metadata` table function
+
+The `parquet_metadata` table function can be used to inspect detailed metadata
+about a parquet file such as statistics, sizes, and other information. This can
+be helpful to understand how parquet files are structured.
+
+For example, to see information about the `"WatchID"` column in the
+`hits.parquet` file, you can use:
+
+```sql
+SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size
+FROM parquet_metadata('hits.parquet')
+WHERE path_in_schema = '"WatchID"'
+LIMIT 3;
+
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| "WatchID" | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 |
+| "WatchID" | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 |
+| "WatchID" | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+3 rows in set. Query took 0.053 seconds.
+```
+
+The returned table has the following columns for each row for each column chunk
+in the file. Please refer to the [Parquet Documentation] for more information.
+
+[parquet documentation]: https://parquet.apache.org/
+
+| column_name | data_type | Description |
+| ----------------------- | --------- | --------------------------------------------------------------------------------------------------- |
+| filename | Utf8 | Name of the file |
+| row_group_id | Int64 | Row group index the column chunk belongs to |
+| row_group_num_rows | Int64 | Count of rows stored in the row group |
+| row_group_num_columns | Int64 | Total number of columns in the row group (same for all row groups) |
+| row_group_bytes | Int64 | Number of bytes used to store the row group (not including metadata) |
+| column_id | Int64 | ID of the column |
+| file_offset | Int64 | Offset within the file that this column chunk's data begins |
+| num_values | Int64 | Total number of values in this column chunk |
+| path_in_schema | Utf8 | "Path" (column name) of the column chunk in the schema |
+| type | Utf8 | Parquet data type of the column chunk |
+| stats_min | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
+| stats_max | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
Review Comment:
```suggestion
| stats_max | Utf8 | The maximum value for this column chunk, if stored in the statistics, cast to a string |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Document parquet_metadata function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852#discussion_r1451592497
##########
docs/source/user-guide/cli.md:
##########
@@ -425,6 +425,13 @@ Available commands inside DataFusion CLI are:
> \h function
```
+## Supported SQL
+
+In addition to the normal [SQL supported in DataFusion], `datatfusion-cli` also
Review Comment:
```suggestion
In addition to the normal [SQL supported in DataFusion], `datafusion-cli` also
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Document `parquet_metadata` function [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852#discussion_r1451729537
##########
docs/source/user-guide/cli.md:
##########
@@ -467,6 +474,66 @@ Available commands inside DataFusion CLI are:
> SET datafusion.execution.batch_size to 1024;
```
+- `parquet_metadata` table function
+
+The `parquet_metadata` table function can be used to inspect detailed metadata
+about a parquet file such as statistics, sizes, and other information. This can
+be helpful to understand how parquet files are structured.
+
+For example, to see information about the `"WatchID"` column in the
+`hits.parquet` file, you can use:
+
+```sql
+SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size
+FROM parquet_metadata('hits.parquet')
+WHERE path_in_schema = '"WatchID"'
+LIMIT 3;
+
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| "WatchID" | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 |
+| "WatchID" | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 |
+| "WatchID" | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+3 rows in set. Query took 0.053 seconds.
+```
+
+The returned table has the following columns for each row for each column chunk
+in the file. Please refer to the [Parquet Documentation] for more information.
+
+[parquet documentation]: https://parquet.apache.org/
+
+| column_name | data_type | Description |
+| ----------------------- | --------- | --------------------------------------------------------------------------------------------------- |
+| filename | Utf8 | Name of the file |
+| row_group_id | Int64 | Row group index the column chunk belongs to |
+| row_group_num_rows | Int64 | Count of rows stored in the row group |
+| row_group_num_columns | Int64 | Total number of columns in the row group (same for all row groups) |
+| row_group_bytes | Int64 | Number of bytes used to store the row group (not including metadata) |
+| column_id | Int64 | ID of the column |
+| file_offset | Int64 | Offset within the file that this column chunk's data begins |
+| num_values | Int64 | Total number of values in this column chunk |
+| path_in_schema | Utf8 | "Path" (column name) of the column chunk in the schema |
+| type | Utf8 | Parquet data type of the column chunk |
+| stats_min | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
+| stats_max | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
+| stats_null_count | Int64 | Number of null values in this column chunk, if stored in the statistics |
+| stats_distinct_count | Int64 | Number of distinct values in this column chunk, if stored in the statistics |
+| stats_min_value | Utf8 | Same as `stats_min` |
Review Comment:
I don't honestly know why the seemingly duplicated columns are present. It was done initially to mirror duckdb which has them. Maybe we should investigate the reason why 🤔
```sql
D create table foo as select * from parquet_metadata('./benchmarks/data/hits.parquet');
D describe table foo;
┌─────────────────────────┬─────────────┬─────────┬─────────┬─────────┬─────────┐
│ column_name │ column_type │ null │ key │ default │ extra │
│ varchar │ varchar │ varchar │ varchar │ varchar │ varchar │
├─────────────────────────┼─────────────┼─────────┼─────────┼─────────┼─────────┤
│ file_name │ VARCHAR │ YES │ │ │ │
│ row_group_id │ BIGINT │ YES │ │ │ │
│ row_group_num_rows │ BIGINT │ YES │ │ │ │
│ row_group_num_columns │ BIGINT │ YES │ │ │ │
│ row_group_bytes │ BIGINT │ YES │ │ │ │
│ column_id │ BIGINT │ YES │ │ │ │
│ file_offset │ BIGINT │ YES │ │ │ │
│ num_values │ BIGINT │ YES │ │ │ │
│ path_in_schema │ VARCHAR │ YES │ │ │ │
│ type │ VARCHAR │ YES │ │ │ │
│ stats_min │ VARCHAR │ YES │ │ │ │
│ stats_max │ VARCHAR │ YES │ │ │ │
│ stats_null_count │ BIGINT │ YES │ │ │ │
│ stats_distinct_count │ BIGINT │ YES │ │ │ │
│ stats_min_value │ VARCHAR │ YES │ │ │ │
│ stats_max_value │ VARCHAR │ YES │ │ │ │
│ compression │ VARCHAR │ YES │ │ │ │
│ encodings │ VARCHAR │ YES │ │ │ │
│ index_page_offset │ BIGINT │ YES │ │ │ │
│ dictionary_page_offset │ BIGINT │ YES │ │ │ │
│ data_page_offset │ BIGINT │ YES │ │ │ │
│ total_compressed_size │ BIGINT │ YES │ │ │ │
│ total_uncompressed_size │ BIGINT │ YES │ │ │ │
├─────────────────────────┴─────────────┴─────────┴─────────┴─────────┴─────────┤
│ 23 rows 6 columns │
└───────────────────────────────────────────────────────────────────────────────┘
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Document parquet_metadata function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852#discussion_r1451592612
##########
docs/source/user-guide/cli.md:
##########
@@ -467,6 +474,66 @@ Available commands inside DataFusion CLI are:
> SET datafusion.execution.batch_size to 1024;
```
+- `parquet_metadata` table function
+
+The `parquet_metadata` table function can be used to inspect detailed metadata
+about a parquet file such as statistics, sizes, and other information. This can
+be helpful to understand how parquet files are structured.
+
+For example, to see information about the `"WatchID"` column in the
+`hits.parquet` file, you can use:
+
+```sql
+SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size
+FROM parquet_metadata('hits.parquet')
+WHERE path_in_schema = '"WatchID"'
+LIMIT 3;
+
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+| "WatchID" | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 |
+| "WatchID" | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 |
+| "WatchID" | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 |
++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+
+3 rows in set. Query took 0.053 seconds.
+```
+
+The returned table has the following columns for each row for each column chunk
+in the file. Please refer to the [Parquet Documentation] for more information.
+
+[parquet documentation]: https://parquet.apache.org/
+
+| column_name | data_type | Description |
+| ----------------------- | --------- | --------------------------------------------------------------------------------------------------- |
+| filename | Utf8 | Name of the file |
+| row_group_id | Int64 | Row group index the column chunk belongs to |
+| row_group_num_rows | Int64 | Count of rows stored in the row group |
+| row_group_num_columns | Int64 | Total number of columns in the row group (same for all row groups) |
+| row_group_bytes | Int64 | Number of bytes used to store the row group (not including metadata) |
+| column_id | Int64 | ID of the column |
+| file_offset | Int64 | Offset within the file that this column chunk's data begins |
+| num_values | Int64 | Total number of values in this column chunk |
+| path_in_schema | Utf8 | "Path" (column name) of the column chunk in the schema |
+| type | Utf8 | Parquet data type of the column chunk |
+| stats_min | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
+| stats_max | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string |
+| stats_null_count | Int64 | Number of null values in this column chunk, if stored in the statistics |
+| stats_distinct_count | Int64 | Number of distinct values in this column chunk, if stored in the statistics |
+| stats_min_value | Utf8 | Same as `stats_min` |
Review Comment:
wondering if this duplicated fields needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Document parquet_metadata function [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852#discussion_r1451502105
##########
docs/source/user-guide/cli.md:
##########
@@ -191,7 +191,7 @@ DataFusion CLI v16.0.0
2 rows in set. Query took 0.007 seconds.
```
-## Creating external tables
+## Creating External Tables
Review Comment:
drive by cleanup -- the other headings are capitalized so it seemed strange that this one was not
##########
docs/source/user-guide/cli.md:
##########
@@ -467,6 +474,66 @@ Available commands inside DataFusion CLI are:
> SET datafusion.execution.batch_size to 1024;
```
+- `parquet_metadata` table function
Review Comment:
Here is what this looks like rendered:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: Document `parquet_metadata` function [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #8852:
URL: https://github.com/apache/arrow-datafusion/pull/8852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org