You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by bl...@apache.org on 2020/04/23 21:03:23 UTC
[incubator-iceberg] branch master updated: Spec: Add field-id to
partition fields (#957)
This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-iceberg.git
The following commit(s) were added to refs/heads/master by this push:
new 0da31db Spec: Add field-id to partition fields (#957)
0da31db is described below
commit 0da31dbcb9ebefbdd8cc8124a1f8dd8ac87528a5
Author: jun-he <ju...@users.noreply.github.com>
AuthorDate: Thu Apr 23 14:03:13 2020 -0700
Spec: Add field-id to partition fields (#957)
---
site/docs/spec.md | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/site/docs/spec.md b/site/docs/spec.md
index 12c1b4b..9940af8 100644
--- a/site/docs/spec.md
+++ b/site/docs/spec.md
@@ -150,6 +150,7 @@ Data files are stored in manifests with a tuple of partition values that are use
Tables are configured with a **partition spec** that defines how to produce a tuple of partition values from a record. A partition spec has a list of fields that consist of:
* A **source column id** from the table’s schema
+* A **partition field id** that is used to identify a partition field, which is unique within a partition spec. In v2 table metadata, it will be unique across all partition specs.
* A **transform** that is applied to the source column to produce a partition value
* A **partition name**
@@ -628,7 +629,7 @@ Partition specs are serialized as a JSON object with the following fields:
|Field|JSON representation|Example|
|--- |--- |--- |
|**`spec-id`**|`JSON int`|`0`|
-|**`fields`**|`JSON list: [`<br /> `<partition field JSON>,`<br /> `...`<br />`]`|`[ {`<br /> `"source-id": 4,`<br /> `"name": "ts_day",`<br /> `"transform": "day"`<br />`}, {`<br /> `"source-id": 1,`<br /> `"name": "id_bucket",`<br /> `"transform": "bucket[16]"`<br />`} ]`|
+|**`fields`**|`JSON list: [`<br /> `<partition field JSON>,`<br /> `...`<br />`]`|`[ {`<br /> `"source-id": 4,`<br /> `"field-id": 1000,`<br /> `"name": "ts_day",`<br /> `"transform": "day"`<br />`}, {`<br /> `"source-id": 1,`<br /> `"field-id": 1001,`<br /> `"name": "id_bucket",`<br /> `"transform": "bucket[16]"`<br />`} ]`|
Each partition field in the fields list is stored as an object. See the table for more detail:
@@ -641,7 +642,7 @@ Each partition field in the fields list is stored as an object. See the table fo
|**`month`**|`JSON string: "month"`|`"month"`|
|**`day`**|`JSON string: "day"`|`"day"`|
|**`hour`**|`JSON string: "hour"`|`"hour"`|
-|**`Partition Field`**|`JSON object: {`<br /> `"source-id": <id int>,`<br /> `"name": <name string>,`<br /> `"transform": <transform JSON>`<br />`}`|`{`<br /> `"source-id": 1,`<br /> `"name": "id_bucket",`<br /> `"transform": "bucket[16]"`<br />`}`|
+|**`Partition Field`**|`JSON object: {`<br /> `"source-id": <id int>,`<br /> `"field-id": <field id int>,`<br /> `"name": <name string>,`<br /> `"transform": <transform JSON>`<br />`}`|`{`<br /> `"source-id": 1,`<br /> `"field-id": 1000,`<br /> `"name": "id_bucket",`<br /> `"transform": "bucket[16]"`<br />`}`|
In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec.