You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/27 20:45:47 UTC

[GitHub] [iceberg] rdblue commented on a change in pull request #3482: spark: supplement spark-queries.md doc

rdblue commented on a change in pull request #3482:
URL: https://github.com/apache/iceberg/pull/3482#discussion_r835960240



##########
File path: site/docs/spark-queries.md
##########
@@ -220,27 +218,140 @@ SELECT * FROM prod.db.table.files
 
 ### Manifests
 
-To show a table's file manifests and each file's metadata, run:
+To show a table's current file manifests and each file's metadata, run:
 
 ```sql
 SELECT * FROM prod.db.table.manifests
 ```
-```text
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
-| path                                                                 | length | partition_spec_id | added_snapshot_id   | added_data_files_count | existing_data_files_count | deleted_data_files_count | partition_summaries                  |
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
+
+<div class="markdown-table-container" markdown="block">
+| path | length | partition_spec_id | added_snapshot_id | added_data_files_count | existing_data_files_count | deleted_data_files_count | partition_summaries |
+| -- | -- | -- | -- | -- | -- | -- | -- |
 | s3://.../table/metadata/45b5290b-ee61-4788-b324-b1e2735c0e10-m0.avro | 4479   | 0                 | 6668963634911763636 | 8                      | 0                         | 0                        | [[false,null,2019-05-13,2019-05-15]] |
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
+</div>
+
+!!! Note
+    1. Fields within `partition_summaries` column of the manifests table correspond to `field_summary` structs within [manifest list](./spec.md#manifest-lists), with the following order: 
+        - `contains_null`
+        - `contains_nan`
+        - `lower_bound`
+        - `upper_bound`
+    2. `contains_nan` could return null, which indicates that this information is not available from files' metadata. 
+        This usually occurs when reading from V1 table, where `contains_nan` is not populated.
+
+### Partitions
+
+To show a table's current partitions
+
+```sql
+SELECT * FROM prod.db.table.partitions
 ```
 
-Note: 
-1. Fields within `partition_summaries` column of the manifests table correspond to `field_summary` structs within [manifest list](./spec.md#manifest-lists), with the following order: 
-   - `contains_null`
-   - `contains_nan`
-   - `lower_bound`
-   - `upper_bound`
-2. `contains_nan` could return null, which indicates that this information is not available from files' metadata. 
-   This usually occurs when reading from V1 table, where `contains_nan` is not populated. 
+<div class="markdown-table-container" markdown="block">
+| partition | record_count | file_count |
+| -- | -- | -- |
+|  {20211001, 11}|           1|         1|
+|  {20211002, 11}|           1|         1|
+|  {20211001, 10}|           1|         1|
+|  {20211002, 10}|           1|         1|
+</div>
+
+### All Metadata Tables
+
+These tables are unions of the metadata tables that are specific to a snapshot.
+
+For example, one table has two snapshots, that's mean a snapshot corresponding to a metadata table, and the "all" metadata tables will contain the results of these two metadata tables.
+
+!!! Warning
+    The "all" metadata tables may produce more than one row per data file or manifest file because metadata files may be used in more than one table snapshot.
+
+#### All Data Files
+
+To show a table's valid data files and each file's metadata, run:
+
+```sql
+SELECT * FROM prod.db.table.all_data_files
+```
+
+<div class="markdown-table-container" markdown="block">
+| content | file_path | file_format | partition | record_count | file_size_in_bytes | column_sizes| value_counts | null_value_counts | nan_value_counts| lower_bounds| upper_bounds|key_metadata|split_offsets|equality_ids|sort_order_id|
+| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
+|      0|s3:.../dt=20210102/xxx.parquet|    PARQUET|{20210102}|          14|              2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}|  {1 -> 0, 2 -> 0}|              {}|{1 -> 1, 2 -> 20210102}|{1 -> 2, 2 -> 20210102}|        null|          [4]|        null|            0|
+|      0|s3:.../dt=20210103/xxx.parquet|    PARQUET|{20210103}|          14|              2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}|  {1 -> 0, 2 -> 0}|              {}|{1 -> 1, 2 -> 20210103}|{1 -> 3, 2 -> 20210103}|        null|          [4]|        null|            0|
+|      0|s3:.../dt=20210104/xxx.parquet|    PARQUET|{20210104}|          14|              2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}|  {1 -> 0, 2 -> 0}|              {}|{1 -> 1, 2 -> 20210104}|{1 -> 3, 2 -> 20210104}|        null|          [4]|        null|            0|
+</div>
+
+#### All Manifests
+
+To show a table's valid file manifests and each file's metadata, run:
+
+```sql
+SELECT * FROM prod.db.table.all_manifests
+```
+
+<div class="markdown-table-container" markdown="block">
+| path | length | partition_spec_id | added_snapshot_id | added_data_files_count | existing_data_files_count | deleted_data_files_count| partition_summaries|
+| -- | -- | -- | -- | -- | -- | -- | -- |
+| s3:/.../table/data/dt=20210102/xxx.parquet | 6376 | 0 | 6272782676904868561 | 2 | 0 | 0 |[{false, false, 10, 11}, {false, false, 20210101, 20210101}]|
+</div>
+
+!!! Note
+    1. A valid manifest file is one that is referenced from any snapshot currently tracked by the table.
+    2. Fields within `partition_summaries` column of the manifests table correspond to `field_summary` structs within [manifest list](./spec.md#manifest-lists), with the following order:
+        - `contains_null`
+        - `contains_nan`
+        - `lower_bound`
+        - `upper_bound`
+    3. `contains_nan` could return null, which indicates that this information is not available from files' metadata.
+        This usually occurs when reading from V1 table, where `contains_nan` is not populated.
+
+### Advanced Usage

Review comment:
       I think to unblock this PR, we should just remove it. I don't think we need it.
   
   I'm okay with adding it eventually, but it is a very low priority to figure out how to document it correctly.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org