You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/01/09 01:19:45 UTC
[GitHub] [iceberg] yyanyy opened a new pull request #2055: add sort order to spec
yyanyy opened a new pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055
Add sort order descriptions to the spec
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r555258356
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,24 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users could sort their data within partitions by columns to gain performance. The information on how the data is sorted could be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. This default could be overridden per file basis if the file is sorted differently, such as if the engine is incapable of ensure ordering of the data on write, the generated files should be annotated with sort order id 0 (unsorted).
+
+Note that only data files and equality delete files could have sort order. [Position deletes](#position-delete-files) should not have sort order, since they have their own sorting requirements.
Review comment:
I would rather not say "should not have a sort order". Instead, this should note that position deletes are required to be sorted by file and position. The manifest should not be written with an order ID for position delete files, and readers must ignore the field for those files. This should probably be in the manifests section instead of here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r562863496
##########
File path: site/docs/spec.md
##########
@@ -831,6 +852,29 @@ Each partition field in the fields list is stored as an object. See the table fo
In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec.
+### Sort Orders
+
+Sort orders are serialized as a list of JSON object, each of which contains the following fields:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`order-id`**|`JSON int`|`1`|
+|**`fields`**|`JSON list: [`<br /> `<sort field JSON>,`<br /> `...`<br />`]`|`[ {`<br /> ` "transform": "identity",`<br /> ` "source-id": 2,`<br /> ` "direction": "asc",`<br /> ` "null-order": "nulls-first"`<br /> `}, {`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`} ]`|
+
+Each sort field in the fields list is stored as an object with the following properties:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`Sort Field`**|`JSON object: {`<br /> `"transform": <transform JSON>,`<br /> `"source-id": <source id int>,`<br /> `"direction": <direction string>,`<br /> `"null-order": <null-order string>`<br />`}`|`{`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`}`|
+
+The following table describes the possible values for the some of the field within sort field:
+
+|Field|JSON representation|Possible values|
+|--- |--- |--- |
+|**`direction`**|`JSON string`|`"asc", "desc"`|
+|**`null-order`**|`JSON string`|`"nulls-first", "nulls-last"`|
Review comment:
A bit more on this: the IEEE 754 spec is quoted in this Rust issue: https://github.com/rust-lang/rust/issues/5585
It confirms that -NaN should sort before all values, and +NaN should sort after all values. So we can note in the doc that floating point ordering should follow the IEEE 754 _totalOrder_ predicate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r555355191
##########
File path: site/docs/spec.md
##########
@@ -831,6 +852,29 @@ Each partition field in the fields list is stored as an object. See the table fo
In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec.
+### Sort Orders
+
+Sort orders are serialized as a list of JSON object, each of which contains the following fields:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`order-id`**|`JSON int`|`1`|
+|**`fields`**|`JSON list: [`<br /> `<sort field JSON>,`<br /> `...`<br />`]`|`[ {`<br /> ` "transform": "identity",`<br /> ` "source-id": 2,`<br /> ` "direction": "asc",`<br /> ` "null-order": "nulls-first"`<br /> `}, {`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`} ]`|
+
+Each sort field in the fields list is stored as an object with the following properties:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`Sort Field`**|`JSON object: {`<br /> `"transform": <transform JSON>,`<br /> `"source-id": <source id int>,`<br /> `"direction": <direction string>,`<br /> `"null-order": <null-order string>`<br />`}`|`{`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`}`|
+
+The following table describes the possible values for the some of the field within sort field:
+
+|Field|JSON representation|Possible values|
+|--- |--- |--- |
+|**`direction`**|`JSON string`|`"asc", "desc"`|
+|**`null-order`**|`JSON string`|`"nulls-first", "nulls-last"`|
Review comment:
how is NaN ordered? In java, `Double.NaN` is considered to be equal to itself and greater than all other double values including `Double.POSITIVE_INFINITY`. If this is the definition we follow, we should document it here. Otherwise something similar like a `NaNOrder` class should be defined in sort order.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r555257207
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,24 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users could sort their data within partitions by columns to gain performance. The information on how the data is sorted could be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. This default could be overridden per file basis if the file is sorted differently, such as if the engine is incapable of ensure ordering of the data on write, the generated files should be annotated with sort order id 0 (unsorted).
Review comment:
I think that it is a little unclear to say the "default could be overridden per file" because that makes it sound as though the default applies to files that do not have a sort order. The spec should specify in the manifests section that if the order ID is not present or unknown, then the order is assumed to be unsorted.
Here, I think you want to clarify what the default sort order is used for: writers _should_ use the default order, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] findepi commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
findepi commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r679108528
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,24 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons.
Review comment:
if you want to tinker with this further, you can find https://github.com/findepi/urandom-bits/blob/417d908ebf5ef43f2d0c5187a5d8914b0139d8f6/src/main/java/findepi/java/NanOrder.java a useful starting point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559030487
##########
File path: site/docs/spec.md
##########
@@ -305,11 +321,14 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo
| _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Implementation-specific key metadata for encryption |
| _optional_ | _optional_ | **`132 split_offsets`** | `list<133: long>` | Split offsets for the data file. For example, all row group offsets in a Parquet file. Must be sorted ascending |
| | _optional_ | **`135 equality_ids`** | `list<136: int>` | Field ids used to determine row equality in equality delete files. Required when `content=2` and should be null otherwise. Fields with ids listed in this column must be present in the delete file |
+| _optional_ | _optional_ | **`140 sort_order_id`** | `int` | ID representing sort order for this file [2]. |
Notes:
1. Single-value serialization for lower and upper bounds is detailed in Appendix D.
+2. If sort order ID is missing or unknown, then the order is assumed to be unsorted. Only data files and equality delete files could have valid sort orders, and [position deletes](#position-delete-files) are required to be sorted by file and position. The manifest should not be written with an order ID for position delete files, and readers must ignore this field for those files.
Review comment:
How about "only data files and equality delete files should be written with a non-null order id. "Position deletes are required to be sorted by file and position, not a table order, and should set sort order id to null. Readers must ignore sort order id for position delete files.".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] findepi commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
findepi commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r679689946
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,24 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons.
Review comment:
@yyanyy you're right -- from representation perspective, NaN value still has a sign bit that can be set or unset, but this indeed has no meaning, at least in Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#issuecomment-765643967
Merged! Thanks @yyanyy! And thanks to @jackye1995 for helping to review!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] findepi commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
findepi commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r679088821
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,24 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons.
Review comment:
The -NaN is a bit ambiguous:
- it can be read as a result of applying unary minus to a NaN value. In Java, `-Double.NaN` is not distinguishable from `Double.NaN` (is exact same value bitwise).
- my JVM returns true from `Double.doubleToRawLongBits(-Double.NaN) == Double.doubleToRawLongBits(Double.NaN)`
- it can be read as a "a IEEE 754 NaN value that has a sign bit set". For exampe
- for example, in my JVM, `Double.longBitsToDouble(0xfff8000000000000L)` is such value, if I read this correctly. and so, `Double.isNaN(Double.longBitsToDouble(0xfff8000000000000L))` is true (while also `Double.doubleToRawLongBits(Double.longBitsToDouble(0xfff8000000000000L)) != Double.doubleToRawLongBits(Double.NaN)` is true, i.e. this expression constructs a NaN value that;s bitwise distinguishable from `Double.NaN`)
In both cases however, "Java floating-point types comparisons" seems to sort all NaN values as peers, and "greater" than positive infinity:
```
System.out.println(Double.compare(Double.POSITIVE_INFINITY, Double.NaN)); // -1
System.out.println(Double.compare(Double.NaN, -Double.NaN)); // 0
System.out.println(Double.compare(Double.NaN, Double.longBitsToDouble(0xfff8000000000000L))); // 0
System.out.println(Double.compare(Double.NaN, Double.longBitsToDouble(0xfff8000012340000L))); // 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559029224
##########
File path: site/docs/spec.md
##########
@@ -831,6 +852,29 @@ Each partition field in the fields list is stored as an object. See the table fo
In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec.
+### Sort Orders
+
+Sort orders are serialized as a list of JSON object, each of which contains the following fields:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`order-id`**|`JSON int`|`1`|
+|**`fields`**|`JSON list: [`<br /> `<sort field JSON>,`<br /> `...`<br />`]`|`[ {`<br /> ` "transform": "identity",`<br /> ` "source-id": 2,`<br /> ` "direction": "asc",`<br /> ` "null-order": "nulls-first"`<br /> `}, {`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`} ]`|
+
+Each sort field in the fields list is stored as an object with the following properties:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`Sort Field`**|`JSON object: {`<br /> `"transform": <transform JSON>,`<br /> `"source-id": <source id int>,`<br /> `"direction": <direction string>,`<br /> `"null-order": <null-order string>`<br />`}`|`{`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`}`|
+
+The following table describes the possible values for the some of the field within sort field:
+
+|Field|JSON representation|Possible values|
+|--- |--- |--- |
+|**`direction`**|`JSON string`|`"asc", "desc"`|
+|**`null-order`**|`JSON string`|`"nulls-first", "nulls-last"`|
Review comment:
The `Float.compare` implementation in Java will sort `NaN` larger than any other value and `-NaN` smaller than any other value. I think we should go with that.
```java
public static int compare(float f1, float f2) {
if (f1 < f2)
return -1; // Neither val is NaN, thisVal is smaller
if (f1 > f2)
return 1; // Neither val is NaN, thisVal is larger
// Cannot use floatToRawIntBits because of possibility of NaNs.
int thisBits = Float.floatToIntBits(f1);
int anotherBits = Float.floatToIntBits(f2);
return (thisBits == anotherBits ? 0 : // Values are equal
(thisBits < anotherBits ? -1 : // (-0.0, 0.0) or (!NaN, NaN)
1)); // (0.0, -0.0) or (NaN, !NaN)
}
```
I think that produces: -NaN < -Infinity < -value < -0 < 0 < value < Infinity < NaN
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559031465
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,22 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users could sort their data within partitions by columns to gain performance. The information on how the data is sorted could be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes.
Review comment:
Looks good!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559026494
##########
File path: site/docs/spec.md
##########
@@ -480,6 +499,8 @@ Table metadata consists of the following fields:
| _optional_ | _optional_ | **`current-snapshot-id`**| `long` ID of the current table snapshot. |
| _optional_ | _optional_ | **`snapshots`**| A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. |
| _optional_ | _optional_ | **`snapshot-log`**| A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed. |
+| _optional_ | _optional_ | **`sort-orders`**| A list of sort orders, stored as full sort order objects. |
+| _optional_ | _optional_ | **`default-sort-order-id`**| Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files. |
Review comment:
I think that we need to clarify this a bit more. Sort order is _optional_ when v2 is reading because v1 metadata is still valid. But a v2 writer must always write the `sort-orders` and `default-sort-order-id` fields. I think we need to change this to _required_ for v2 and make sure there is text stating that these requirements are for writers and that v2 readers must accept all valid v1 metadata.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r562865065
##########
File path: site/docs/spec.md
##########
@@ -480,6 +499,8 @@ Table metadata consists of the following fields:
| _optional_ | _optional_ | **`current-snapshot-id`**| `long` ID of the current table snapshot. |
| _optional_ | _optional_ | **`snapshots`**| A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. |
| _optional_ | _optional_ | **`snapshot-log`**| A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed. |
+| _optional_ | _optional_ | **`sort-orders`**| A list of sort orders, stored as full sort order objects. |
+| _optional_ | _optional_ | **`default-sort-order-id`**| Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files. |
Review comment:
Yes, since that section is intended to clarify, not specify. That doesn't need to be part of this PR, though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#issuecomment-761625324
Thanks @yyanyy! This looks very close to being ready.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
yyanyy commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r679522667
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,24 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons.
Review comment:
Thanks for the input! You are right that in Java `-Double.NaN` equals `Double.NaN` so I think we should drop the mentioning of `-NaN` in spec. For the interpretation of having the sign bit set, I did some quick reading and it seems like NaN can be represented by a lot of ways (from [here](https://stackoverflow.com/a/2154512) it says anything that looks like `x111 1111 1axx xxxx xxxx xxxx xxxx xxxx` where x means anything would be NaN), and the sign bit for NaN seems to have [no meaning](https://en.wikipedia.org/wiki/IEEE_754). In this case I think we probably want to avoid mentioning `-NaN` in spec completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
yyanyy commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r555453177
##########
File path: site/docs/spec.md
##########
@@ -831,6 +852,29 @@ Each partition field in the fields list is stored as an object. See the table fo
In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec.
+### Sort Orders
+
+Sort orders are serialized as a list of JSON object, each of which contains the following fields:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`order-id`**|`JSON int`|`1`|
+|**`fields`**|`JSON list: [`<br /> `<sort field JSON>,`<br /> `...`<br />`]`|`[ {`<br /> ` "transform": "identity",`<br /> ` "source-id": 2,`<br /> ` "direction": "asc",`<br /> ` "null-order": "nulls-first"`<br /> `}, {`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`} ]`|
+
+Each sort field in the fields list is stored as an object with the following properties:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`Sort Field`**|`JSON object: {`<br /> `"transform": <transform JSON>,`<br /> `"source-id": <source id int>,`<br /> `"direction": <direction string>,`<br /> `"null-order": <null-order string>`<br />`}`|`{`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`}`|
+
+The following table describes the possible values for the some of the field within sort field:
+
+|Field|JSON representation|Possible values|
+|--- |--- |--- |
+|**`direction`**|`JSON string`|`"asc", "desc"`|
+|**`null-order`**|`JSON string`|`"nulls-first", "nulls-last"`|
Review comment:
Thanks, I think this is a good point I missed. I'm hoping to treat NaN as the largest value by default, but I'm not familiar with engines' implementations to say if it's something that could easily be achieved without much effort. @rdblue @aokolnychyi Do you have any recommendation to this? Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] findepi commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
findepi commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r679112476
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,24 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users can sort their data within partitions by columns to gain performance. The information on how the data is sorted can be declared per data or delete file, by a **sort order**.
+
+A sort order is defined by an sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data. Each sort field consists of:
+
+* A **source column id** from the table's schema
+* A **transform** that is used to produce values to be sorted on from the source column. This is the same transform as described in [partition transforms](#partition-transforms).
+* A **sort direction**, that can only be either `asc` or `desc`
+* A **null order** that describes the order of null values when sorted. Can only be either `nulls-first` or `nulls-last`
+
+Order id `0` is reserved for the unsorted order.
+
+Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons.
Review comment:
if your thinking matches mine, here is the PR https://github.com/apache/iceberg/pull/2891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559026494
##########
File path: site/docs/spec.md
##########
@@ -480,6 +499,8 @@ Table metadata consists of the following fields:
| _optional_ | _optional_ | **`current-snapshot-id`**| `long` ID of the current table snapshot. |
| _optional_ | _optional_ | **`snapshots`**| A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. |
| _optional_ | _optional_ | **`snapshot-log`**| A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed. |
+| _optional_ | _optional_ | **`sort-orders`**| A list of sort orders, stored as full sort order objects. |
+| _optional_ | _optional_ | **`default-sort-order-id`**| Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files. |
Review comment:
I think that we need to clarify this a bit more. Sort order is _optional_ in v2 because v1 metadata is still valid. But a v2 writer must always write the `sort-orders` and `default-sort-order-id` fields. I think we need to change this to _required_ for v2 and make sure there is text stating that these requirements are for writers and that v2 readers must accept all valid v1 metadata.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559030487
##########
File path: site/docs/spec.md
##########
@@ -305,11 +321,14 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo
| _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Implementation-specific key metadata for encryption |
| _optional_ | _optional_ | **`132 split_offsets`** | `list<133: long>` | Split offsets for the data file. For example, all row group offsets in a Parquet file. Must be sorted ascending |
| | _optional_ | **`135 equality_ids`** | `list<136: int>` | Field ids used to determine row equality in equality delete files. Required when `content=2` and should be null otherwise. Fields with ids listed in this column must be present in the delete file |
+| _optional_ | _optional_ | **`140 sort_order_id`** | `int` | ID representing sort order for this file [2]. |
Notes:
1. Single-value serialization for lower and upper bounds is detailed in Appendix D.
+2. If sort order ID is missing or unknown, then the order is assumed to be unsorted. Only data files and equality delete files could have valid sort orders, and [position deletes](#position-delete-files) are required to be sorted by file and position. The manifest should not be written with an order ID for position delete files, and readers must ignore this field for those files.
Review comment:
How about "only data files and equality delete files should be written with a non-null order id. "Position deletes are required to be sorted by file and position, not a table order, and should set sort order id to null".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559031848
##########
File path: site/docs/spec.md
##########
@@ -831,6 +852,29 @@ Each partition field in the fields list is stored as an object. See the table fo
In some cases partition specs are stored using only the field list instead of the object format that includes the spec ID, like the deprecated `partition-spec` field in table metadata. The object format should be used unless otherwise noted in this spec.
+### Sort Orders
+
+Sort orders are serialized as a list of JSON object, each of which contains the following fields:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`order-id`**|`JSON int`|`1`|
+|**`fields`**|`JSON list: [`<br /> `<sort field JSON>,`<br /> `...`<br />`]`|`[ {`<br /> ` "transform": "identity",`<br /> ` "source-id": 2,`<br /> ` "direction": "asc",`<br /> ` "null-order": "nulls-first"`<br /> `}, {`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`} ]`|
+
+Each sort field in the fields list is stored as an object with the following properties:
+
+|Field|JSON representation|Example|
+|--- |--- |--- |
+|**`Sort Field`**|`JSON object: {`<br /> `"transform": <transform JSON>,`<br /> `"source-id": <source id int>,`<br /> `"direction": <direction string>,`<br /> `"null-order": <null-order string>`<br />`}`|`{`<br /> ` "transform": "bucket[4]",`<br /> ` "source-id": 3,`<br /> ` "direction": "desc",`<br /> ` "null-order": "nulls-last"`<br />`}`|
+
+The following table describes the possible values for the some of the field within sort field:
+
+|Field|JSON representation|Possible values|
+|--- |--- |--- |
+|**`direction`**|`JSON string`|`"asc", "desc"`|
+|**`null-order`**|`JSON string`|`"nulls-first", "nulls-last"`|
Review comment:
Yeah this order sounds good to me
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r559031624
##########
File path: site/docs/spec.md
##########
@@ -254,6 +254,22 @@ Notes:
2. The width, `W`, used to truncate decimal values is applied using the scale of the decimal column to avoid additional (and potentially conflicting) parameters.
+### Sorting
+
+Users could sort their data within partitions by columns to gain performance. The information on how the data is sorted could be declared per data or delete file, by a **sort order**.
Review comment:
Could you change both uses of "could" to "can" in this paragraph? That way there is no implied past tense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r555259707
##########
File path: site/docs/spec.md
##########
@@ -480,6 +499,8 @@ Table metadata consists of the following fields:
| _optional_ | _optional_ | **`current-snapshot-id`**| `long` ID of the current table snapshot. |
| _optional_ | _optional_ | **`snapshots`**| A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. |
| _optional_ | _optional_ | **`snapshot-log`**| A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed. |
+| _optional_ | _optional_ | **`sort-orders`**| A list of sort orders, stored as full sort order objects. |
+| _optional_ | _optional_ | **`default-sort-order-id`**| Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files. |
Review comment:
@aokolnychyi, do you think that sort order information should be optional in v2?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yyanyy commented on a change in pull request #2055: Spec: add sort order to spec
Posted by GitBox <gi...@apache.org>.
yyanyy commented on a change in pull request #2055:
URL: https://github.com/apache/iceberg/pull/2055#discussion_r560604071
##########
File path: site/docs/spec.md
##########
@@ -480,6 +499,8 @@ Table metadata consists of the following fields:
| _optional_ | _optional_ | **`current-snapshot-id`**| `long` ID of the current table snapshot. |
| _optional_ | _optional_ | **`snapshots`**| A list of valid snapshots. Valid snapshots are snapshots for which all data files exist in the file system. A data file must not be deleted from the file system until the last snapshot in which it was listed is garbage collected. |
| _optional_ | _optional_ | **`snapshot-log`**| A list (optional) of timestamp and snapshot ID pairs that encodes changes to the current snapshot for the table. Each time the current-snapshot-id is changed, a new entry should be added with the last-updated-ms and the new current-snapshot-id. When snapshots are expired from the list of valid snapshots, all entries before a snapshot that has expired should be removed. |
+| _optional_ | _optional_ | **`sort-orders`**| A list of sort orders, stored as full sort order objects. |
+| _optional_ | _optional_ | **`default-sort-order-id`**| Default sort order id of the table. Note that this could be used by writers, but is not used when reading because reads use the specs stored in manifest files. |
Review comment:
I think the last sentence in the current [formatv2 section](https://iceberg.apache.org/spec/#version-2) basically coverers the v2 readers accepting valid v1 metadata part. I'll add `sort-orders` and `default-sort-order-id` to the list of this section for now, and do you think if we should move this section/add more explanation about this in other places?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org