You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "wgtmac (via GitHub)" <gi...@apache.org> on 2024/10/30 02:23:12 UTC

[PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

wgtmac opened a new pull request, #466:
URL: https://github.com/apache/parquet-format/pull/466

   ### Rationale for this change
   
   The C++ reader of parquet-cpp is having a hard time to read Parquet file written by parquet-java with `parquet.avro.write-old-list-structure=true` and schema below:
   ```
   optional group a (LIST) {
     repeated group array (LIST) {
       repeated int32 array;
     }
   }
   ```
   
   See https://github.com/apache/arrow/issues/43994 and https://github.com/apache/arrow/pull/43995
   
   ### What changes are included in this PR?
   
   Clarify the rules and add an example.
   
   ### Do these changes have PoC implementations?
   
   Not required
   
   Closes #465
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841875732


##########
LogicalTypes.md:
##########
@@ -670,6 +681,8 @@ optional group array_of_arrays (LIST) {
 
 #### Backward-compatibility rules
 

Review Comment:
   ```suggestion
   
   Modern writers should always produce the 3-level LIST structure shown above.
   However, historically data files have been produced that use different structures to
   represent list-like data, and readers may include compatibility measures to interpret
   them as intended.
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841881096


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   `Rule 4` is only about naming and these are the two options. I think it is fine to have both to be clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2445753052

   BTW, can you also point out the Java code in this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1854389700


##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.

Review Comment:
   "the repetition of a 2-level structure can be" so here means the outer "list-repetition"? Should we denote it as out-most level?



##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures

Review Comment:
   So here following the rule:
   1. Trying to parse as 3-level structure
   2. Trying to parse 2-level legacy
   3. Trying to parse 1-level legacy
   
   ?



##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
+should always be determined by the following rules if they cannot be determined
+as 3-level structures:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field and the repetition of that
+   field is `repeated`, then its type is the element type and elements are

Review Comment:
   I fount the "repeated field" is `repeated` is ambigious here...



##########
LogicalTypes.md:
##########
@@ -609,9 +609,20 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
-Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
-repeated fields, but not both. When using the annotations, no unannotated
-repeated types are allowed.
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}
+```
+
+For all fields in the schema, implementations should use either `LIST` and

Review Comment:
   👍I think this is neccessary, hope implemetation would follow this rule...



##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;

Review Comment:
   Below there is a similiar example with `optional str` after `num`



##########
LogicalTypes.md:
##########
@@ -609,9 +609,20 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
-Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
-repeated fields, but not both. When using the annotations, no unannotated
-repeated types are allowed.
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)

Review Comment:
   nit: `List<Tuple<Integer, String>>` have some way to repr the `String` is nullable in type annotation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1855134279


##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.

Review Comment:
   I think this is clear because we are talking about the whole 2-level structure here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1859240321


##########
LogicalTypes.md:
##########
@@ -684,44 +702,67 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}

Review Comment:
   Again, I think that calling attention to the degenerate cases and documenting them is only going to cause more confusion. The purpose of this originally was to simply document how to interpret data that doesn't match expectations. Now this introduces how a 2-level list looks, which I think increases the possibility that people will misread this and write them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2503276670

   Concretely, if I have to review a PR implementing said compatibility rules, I'd rather have the spec spell out the examples because it makes it easier to check the PR's implementation (and unit tests!). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1823587574


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   I might leave the entire block as it was previously and clarify that:
   1. Without annotatation the type is a single struct/tuple with the element in it.
   2. With a logical type annotation (I think only LIST and MAP apply here).
   
   I also think we need to update the documentation above:
   ```
   The outer-most level must be a group annotated with `LIST` that contains a
     single field named `list`. The repetition of this level must be either
     `optional` or `required` and determines whether the list is nullable.
   ```
   To point to the fact that for backwards compatibility it can also be repeated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2445769036

   Other LGTM but I think it worths issue a disscussion...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1835180625


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.

Review Comment:
   I find this confusing. Why does it need to change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1840215856


##########
LogicalTypes.md:
##########
@@ -684,49 +686,99 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group foo {
+    repeated int32 bar;
+  };
+}
+```
+
+##### 1-level structure without `LIST` annotation
+
+Some existing data does not even have the `LIST` annotation and simply uses
+`repeated` repetition to annotate the element type. For backward-compatibility,
+both the list and elements are `required`.
+
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// Tuple<List<Integer>, List<String>> (non-null list, non-null elements)
+optional group my_list {
+  repeated int32 num;
+  repeated binary str (STRING);
+}
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}
 ```

Review Comment:
   It might be cleaner but not sure if we can safely mix them. Even if we can, it contradicts to the spec. See [here](https://github.com/apache/parquet-format/pull/466#pullrequestreview-2432448521).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1839196589


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.

Review Comment:
   I just happen to see that this section is all about 2-level list so I have added a title to make it explicit.
   
   > The problem is when we need to decide whether a structure is a 2-level or 3-level.
   
   The change aims to exhaust all possible forms of list structures and we can just apply these rules in order to deduce a list type:
   - regular 3-level list
   - 3-level list (w/ different names)
   - 2-level list
   - 1-level list (w/o list annotation)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2503421529

   Ok, please don't use UPPERCASE :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1856207456


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   What would happen if the naming does not satisfy Rule 4? We should give an example of this fallback case.
   
   (I'm also afraid this rule is pointlessly complicated)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1869082194


##########
LogicalTypes.md:
##########
@@ -684,44 +689,58 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
 should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated

Review Comment:
   Redundant with what? We've been explicitly listing examples for all other rules.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1835180625


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.

Review Comment:
   I find this confusing. Why does it need to change?
   
   The problem isn't when there is a 2-level structure like the one in the new example. The problem is when we need to decide whether a structure is a 2-level or 3-level.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1827239798


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,

Review Comment:
   At one point I thought you had rule 3 specify that the repeated group is unannotated. Without that, should rules 3 and 4 be swapped?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1825643095


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   I think we'd better open another PR to discuss list types without annotation. It might be more complicated than the scope of this PR. I'm not sure if it should be allowed by the spec even though we are able to create the schema.
   
   EDIT: after thinking about it again, we may add following words to the end of the Lists section.
   
   > Some existing data does not even have the `LIST` annotation and simply uses
   > `repeated` repetition to annotate the element type. In this case, the element
   > type MUST be a primitive type and both the list and elements are required.
   > 
   > ```
   > // List<Integer> (non-null list, non-null elements)
   > repeated int32 num;
   >
   > // Struct<List<Integer>, List<String>> (non-null list, non-null elements)
   > optional group whatever {
   >   repeated int32 num;
   >   repeated binary str (STRING);
   > }
   > ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1827257237


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,

Review Comment:
   Got it, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2445842558

   I have sent a discussion thread to the dev ML. It would be good if you can take a look. Thanks! @emkornfield @pitrou @gszadovszky @rdblue @etseidl @clairemcginty 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1843418984


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   > Not sure, if we can use the 3-level list for protobuf.
   
   Hmm, what does protobuf have to do with it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1839276100


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.

Review Comment:
   I decide to keep the subtitles as they are pretty clear. It is the exact order to apply when deducing a list type. Please let me know if you have different opinions and I'm happy to change. @rdblue @gszadovszky @emkornfield 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1840476054


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {

Review Comment:
   I believe this is the case where annotated 2-level list is mixed with unannotated repeated field (1-level list). Is this allowed? If not, I can remove the example. Then, is there any concrete example for the last rule (a.k.a. rule 5 above)? @gszadovszky @rdblue 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1856853739


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   @etseidl Yes, I was wrong earlier. I think I have to sleep now :/



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2486365585

   > No, the middle level can never be a LIST-annotated or MAP-annotated 3-level structure, because they cannot be `repeated`.
   
   Ah, ok. Now I understand. That's why the 2-level list needs the ability to be `repeated`, but only as part of a nested list. Thanks again for the clarification @wgtmac. I think all my questions are answered now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1827181829


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,
+   then the element type is a list type with 2-level structure and elements are
+   required.
+5. Otherwise, the repeated field's type is the element type with the repeated
+   field's repetition. Please note that the repeated field here (a group with
+   one field) cannot be `LIST`-annotated or `MAP`-annotated 3-level structure,
+   as such a group's repetition must be `required` or `optional`.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Struct<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+

Review Comment:
   As I have explained in https://lists.apache.org/thread/s6b25j3x26009v054yqjov0f1z49ctqj, the only case left for rule 5 is as below:
   ```
   optional group foo (LIST) {
     repeated group bar {
       repeated TYPE baz;
     };
   }
   ```
   
   It should be resolved to `List<Struct<List<TYPE>>>` if the 1-level structure is allowed, or an invalid case at all.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1829789204


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,
+   then the element type is a list type with 2-level structure and elements are
+   required.
+5. Otherwise, the repeated field's type is the element type with the repeated
+   field's repetition. Please note that the repeated field here (a group with
+   one field) cannot be `LIST`-annotated or `MAP`-annotated 3-level structure,
+   as such a group's repetition must be `required` or `optional`.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Struct<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// List<List<Integer>>
+// Rule 4: nullable outer list with non-null elements
+// Rule 1: non-null inner list with non-null elements
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 5: List<Struct<List<Integer>>> (nullable outer list with non-null elements)
+optional group my_list (LIST) {
+  repeated group foo {
+    repeated int32 bar;
+  };
+}
+```
+
+##### 1-level structure without `LIST` annotation
+
+Some existing data does not even have the `LIST` annotation and simply uses
+`repeated` repetition to annotate the element type. In this case, the element
+type MUST be a primitive type and both the list and elements are required.

Review Comment:
   ```suggestion
   `repeated` repetition to annotate the element type. In this case both the list
   and elements are required.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2483831801

   Ok, so the new Rule 3 is in effect saying that if the child of the repeated middle group is repeated, then because 1-level and LIST/MAP annotated schemas cannot mix, the middle group cannot be the type so the child is the type. The side consequence of this is that the middle group has to have a `LIST` or `MAP` annotation to allow the 3rd level to be `repeated`. 
   
   So is it true then that the `repeated` keyword may only be used for the child of a `LIST` or `MAP` annotated field when those annotations are in use? And if that's true, then a 2-level structure cannot be `repeated` as stated above?
   ```
   In contrast to 3-level structure, the repetition of 2-level structure can be `optional`, `required`, or `repeated`.
   ```
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1842475494


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;

Review Comment:
   `str` and `num` can be optional, but the `element` itself cannot be null (but `my_list` can be empty).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841813821


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   Is it worth listing this example? It's the exact same as the one above, just with a different mid-level group name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1855230005


##########
LogicalTypes.md:
##########
@@ -670,6 +681,13 @@ optional group array_of_arrays (LIST) {
 
 #### Backward-compatibility rules
 
+Modern writers should always produce the 3-level LIST structure shown above.

Review Comment:
   ```suggestion
   New writer implementations should always produce the 3-level LIST structure shown above.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2503273594

   @rdblue @wgtmac I'm not sure I agree with removing the examples. They make it less likely that an implementation would interpret the written rules wrongly, IMO. Maybe we can instead strengthen the wording that these examples show existing legacy schemas, not anything to be written by modern writers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1868965095


##########
LogicalTypes.md:
##########
@@ -684,44 +703,61 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
 should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+WARNING: writers should not produce list types like these examples! They are
+just for the purpose of reading existing data for backward-compatibility.
+
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);

Review Comment:
   For the record: I would expect an example of Rule 5 below?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1859989177


##########
LogicalTypes.md:
##########
@@ -684,44 +702,67 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}

Review Comment:
   Removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1825646221


##########
LogicalTypes.md:
##########
@@ -648,49 +648,66 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+Some existing data does not include the inner element layer, meaning that

Review Comment:
   I have added more explanation according to the ML discussion. I'm not a native speaker, please help me check if these are suitable. Thanks! @rdblue @gszadovszky 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1842484001


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   I tend to agree, but part of me finds it odd that this language will be in `LogicalTypes.md` when without the annotation these are not logical types. But short of a new markdown file I don't know where else to put this, unless there's a section added to `README.md` devoted to how to interpret schemas.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2475057996

   > Does it mean a file schema shall not contain both types of lists or that we shall not use annotated lists inside an unannotated one or vice-versa? @rdblue, WDYT?
   
   I think this is addressed [here](https://github.com/apache/parquet-format/pull/17#discussion_r25807732). The intent was to not allow mixing list types within a schema, so I think all of the above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841811479


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;

Review Comment:
   But this child field could be "optional", right?



##########
LogicalTypes.md:
##########
@@ -670,6 +681,8 @@ optional group array_of_arrays (LIST) {
 
 #### Backward-compatibility rules

Review Comment:
   It is nice that we are including rules for how to interpret existing data, but can we also have guidelines for writers such that we converge on a single rule for future data? It would be nice to reduce the existing variability in the long term.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2502982189

   @rdblue Thanks for your review! I have removed all unnecessary changes. Please take a look again. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1859233788


##########
LogicalTypes.md:
##########
@@ -609,9 +609,20 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
-Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
-repeated fields, but not both. When using the annotations, no unannotated
-repeated types are allowed.
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}
+```
+
+For all fields in the schema, implementations should use either `LIST` and

Review Comment:
   -0 on this change. I don't think this is more clear and I would prefer not to have the churn.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1859997738


##########
LogicalTypes.md:
##########
@@ -684,44 +689,58 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
 should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated

Review Comment:
   I have reverted most of the previous changes and now it should be clear. @etseidl @mapleFU 
   
   To resolve a LIST-annotated group, we should apply rules in order:
   - check if it is a 2-level structure (rule 1 to 3)
   - check if it is a special 2-level structure (rule 4)
   - it is a 3-level structure (rule 5)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2503398615

   Perhaps add them like this? @pitrou 
   ```
   WARNING: WRITERS SHOULD NOT PRODUCE LIST TYPES LIKE THESE EXAMPLES!
   THEY ARE FOR THE PURPOSE OF READING EXISTING DATA ONLY.
   
   // List<Integer> (non-null list, non-null elements)
   repeated int32 num;
   
   // List<Tuple<Integer, String>> (non-null list, non-null elements)
   repeated group my_list {
     required int32 num;
     optional binary str (STRING);
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2519277990

   Will merge this by the end of this week if no objection.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1869077618


##########
LogicalTypes.md:
##########
@@ -684,44 +689,58 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
 should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated

Review Comment:
   @pitrou Regarding to your [question](https://github.com/apache/parquet-format/pull/466#discussion_r1868965095) on rule 5, I think it is redundant to add another one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1839180500


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.

Review Comment:
   Yes, my intention is to differentiate it with the point 3 below which is a struct with a single field. I can remove the redundant statement if it is clear enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1823727500


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   > To point to the fact that for backwards compatibility it can also be repeated.
   
   The original statement is all about the formal three-level encoding. Adding `repeated` to it may further confuse readers. That's why I tried to clarify this in the backward compatibility section.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2449039626

   There is another issue at https://github.com/apache/parquet-format/blob/master/LogicalTypes.md?plain=1#L696-L712.
   
   > `MAP` is used to annotate types that should be interpreted as a map from keys
   > to values. `MAP` must annotate a 3-level structure:
   > 
   > ```
   > <map-repetition> group <name> (MAP) {
   >   repeated group key_value {
   >     required <key-type> key;
   >     <value-repetition> <value-type> value;
   >   }
   > }
   > ```
   > 
   > The outer-most level must be a group annotated with `MAP` that contains a single field named `key_value`. The repetition of this level must be either `optional` or `required` and determines whether the list is nullable.
   
   Does it mean that MAP cannot be nested in two-level encoded LIST? So the 1st one below is valid and the 2nd one is invalid. What should reader implementation do if reading from a Parquet file with the latter schema? @emkornfield @gszadovszky 
   ```
    List<Map<String, String>> in three-level list encoding:
    optional group my_list (LIST) {
      repeated group list {
        required group element (MAP) {
          repeated group key_value {
            required binary key (STRING);
            optional binary value (STRING);
          }
        }
      }
    }
   
    List<Map<String, String>> in two-level list encoding:
    optional group my_list (LIST) {
      repeated group array (MAP) {
        repeated group key_value {
          required binary key (STRING);
          optional binary value (STRING);
        }
      }
    }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1829600414


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,
+   then the element type is a list type with 2-level structure and elements are
+   required.
+5. Otherwise, the repeated field's type is the element type with the repeated
+   field's repetition. Please note that the repeated field here (a group with
+   one field) cannot be `LIST`-annotated or `MAP`-annotated 3-level structure,
+   as such a group's repetition must be `required` or `optional`.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Struct<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// List<List<Integer>>
+// Rule 4: nullable outer list with non-null elements
+// Rule 1: non-null inner list with non-null elements
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  }
+}
+```
+
+##### 1-level structure without `LIST` annotation
+
+Some existing data does not even have the `LIST` annotation and simply uses
+`repeated` repetition to annotate the element type. In this case, the element
+type MUST be a primitive type and both the list and elements are required.

Review Comment:
   https://github.com/apache/parquet-format/pull/17 is an interesting read, and explains why the unannotated version exists (allows direct conversion from protobuf). I _would_ prefer there be a single, canonical way to represent lists, so perhaps we should bring up deprecating the direct form on the ML. I don't know how relevant it is now (and in fact only learned of its existence last week).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1835181019


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,
+   then the element type is a list type with 2-level structure and elements are
+   required.
+5. Otherwise, the repeated field's type is the element type with the repeated
+   field's repetition. Please note that the repeated field here (a group with
+   one field) cannot be `LIST`-annotated or `MAP`-annotated 3-level structure,
+   as such a group's repetition must be `required` or `optional`.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Struct<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)

Review Comment:
   I think this makes the example less clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1840194949


##########
LogicalTypes.md:
##########
@@ -684,49 +686,99 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group foo {
+    repeated int32 bar;
+  };
+}
+```
+
+##### 1-level structure without `LIST` annotation
+
+Some existing data does not even have the `LIST` annotation and simply uses
+`repeated` repetition to annotate the element type. For backward-compatibility,
+both the list and elements are `required`.
+
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// Tuple<List<Integer>, List<String>> (non-null list, non-null elements)
+optional group my_list {
+  repeated int32 num;
+  repeated binary str (STRING);
+}
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}
 ```

Review Comment:
   Good catch! I have moved the example closer to the description. IMO, we should not deprecate it because it is not confusing and actually much cleaner to present a non-null list.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841745716


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {

Review Comment:
   So, it seems it is not allowed. See [here](https://github.com/apache/parquet-format/pull/466#issuecomment-2475057996).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841876784


##########
LogicalTypes.md:
##########
@@ -670,6 +681,8 @@ optional group array_of_arrays (LIST) {
 
 #### Backward-compatibility rules

Review Comment:
   Okay, so I've suggested an addition below to make this extra clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1856847518


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   But now the Rule 5 example is simply a "3-level structure with different field names". It would be `List<String>`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1859998523


##########
LogicalTypes.md:
##########
@@ -684,44 +689,58 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
 should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated

Review Comment:
   I don't want to add an example for rule 5 because it is already at line 685



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #466:
URL: https://github.com/apache/parquet-format/pull/466


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1828608033


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,
+   then the element type is a list type with 2-level structure and elements are
+   required.
+5. Otherwise, the repeated field's type is the element type with the repeated
+   field's repetition. Please note that the repeated field here (a group with
+   one field) cannot be `LIST`-annotated or `MAP`-annotated 3-level structure,
+   as such a group's repetition must be `required` or `optional`.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Struct<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// List<List<Integer>>
+// Rule 4: nullable outer list with non-null elements
+// Rule 1: non-null inner list with non-null elements
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  }
+}
+```
+
+##### 1-level structure without `LIST` annotation
+
+Some existing data does not even have the `LIST` annotation and simply uses
+`repeated` repetition to annotate the element type. In this case, the element
+type MUST be a primitive type and both the list and elements are required.

Review Comment:
   > AFAICT using repeated without the LIST annotation is still supported by the spec
   
   I don't think it is supported by the spec because it is unclear yet. The official list type in the spec is the LIST-annotated group with 3-level structure, which support arbitrary nesting and full capability to specify nullability of each level. The LIST-annotated group with 3-level structure should always be used by writers and others fall into the category of backward compatibility to deal with existing files. A writer can accidentally produce such files does not mean it should be that way.
   
   > Also, I don't think the current [wording](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#nested-types) requires a primitive type
   
   That's true. Let me change this.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1847569585


##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures

Review Comment:
   Explicitly add `if they cannot be determined as 3-level structures` to partially address the comment on how to resolve a list-annotated group in practice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1823567701


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   ```suggestion
   3. If the repeated field is a group (without `LIST` annotation) with one `required` or
   ```
   Not sure if this should be made explicit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1828278839


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,
+   then the element type is a list type with 2-level structure and elements are
+   required.
+5. Otherwise, the repeated field's type is the element type with the repeated
+   field's repetition. Please note that the repeated field here (a group with
+   one field) cannot be `LIST`-annotated or `MAP`-annotated 3-level structure,
+   as such a group's repetition must be `required` or `optional`.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Struct<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// List<List<Integer>>
+// Rule 4: nullable outer list with non-null elements
+// Rule 1: non-null inner list with non-null elements
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  }
+}
+```
+
+##### 1-level structure without `LIST` annotation
+
+Some existing data does not even have the `LIST` annotation and simply uses
+`repeated` repetition to annotate the element type. In this case, the element
+type MUST be a primitive type and both the list and elements are required.

Review Comment:
   I don't know if this belongs in the backwards compatibility section. AFAICT using `repeated` without the `LIST` annotation is still supported by the spec (and some writers can still produce it, including arrow-rs; cc @zeevm).
   
   Also, I don't think the current [wording](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#nested-types) requires a primitive type. The following should be allowed (`List<Struct<Integer, String>> non-null list, non-null elements`):
   ```
   repeated group list_struct {
     required int32 id;
     optional binary val (STRING);
   }
   ```
   In fact, in parquet-testing there's a file `repeated_no_annotation.parquet` with an unannotated `repeated` group:
   ```
   message user {
     REQUIRED INT32 id;
     OPTIONAL group phoneNumbers {
       REPEATED group phone {
         REQUIRED INT64 number;
         OPTIONAL BYTE_ARRAY kind (UTF8);
       }
     }
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1835182164


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,

Review Comment:
   This rule should not be modified with new cases. I think we need to insert a new rule that states if the repeated field is a group with a `repeated` field, the repeated field is the element type because the type cannot be a 3-level list.
   
   The rules should be as independent as possible and I think it makes it more confusing to mix them together like this and like the new #4.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1842551182


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {

Review Comment:
   Looking back at the original wording of the rules, I think what is now rule 3 would have been handled by the old rule 4 (now rule 5), except the added wrinkle of the repeated group being named `array` made it ambiguous. The new rule 3 handles this now, so I'm not sure if rule 5 is reachable any longer.
   
   Rule 1 is for the repeated field not being a group. All other rules apply when the repeated field is a group. If it's a multi-element group, then it's a list of struct (rule 2). If it's a nested 2-level list, then list of list (rule 3). If it's 3-level structure with special naming, it's list of one-tuple (rule 4). What other 1-element group is unhandled now? If the repeated group contains a single `required` or `optional` field, it's a 3-level structure with misnamed parts. The repeated group cannot contain a single `repeated` field since that would be mixing 1-level with 3-level.
   
   Am I missing something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2481322736

   @etseidl I think we are on the same page that current rule 3 has an issue that it does not forbid the inner-most layer to be `repeated`. However, I feel that it is not a good idea to add `would otherwise be a valid 3-level structure as outlined above` to rule 3. The reason is that a normal 3-level structure can simply ignore its synthetic middle layer but the middle layer produces a `OneTuple` in the rule 3, which may also confuse readers. In addition, it is not that straight-forward. So I agree with @rdblue that it would be good to insert a new rule for the new case. IMO the new rule should be placed above the current rule 3 so we are pretty sure that the inner layer cannot be `repeated` in the remaining cases (including the rule 3). The only caveat is that the new rule 5 seems to be unreachable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1847568298


##########
LogicalTypes.md:
##########
@@ -609,9 +609,20 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
-Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
-repeated fields, but not both. When using the annotations, no unannotated
-repeated types are allowed.
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}
+```
+
+For all fields in the schema, implementations should use either `LIST` and

Review Comment:
   Explicitly state that this applies to `all fields in the schema`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1845441650


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {

Review Comment:
   @etseidl Your analysis aligns with mine on the dev ML. I don't think there is any real case in rule 5.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1855486203


##########
LogicalTypes.md:
##########
@@ -684,44 +702,67 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
+should always be determined by the following rules if they cannot be determined
+as 3-level structures:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`

Review Comment:
   The inner-most field is repeated in rule 3 but cannot be repeated in rule 4. In addition, rule 4 refers to a special case produced by legacy thrift and avro writer in the parquet-mr, therefore it must use the exact field name to match.



##########
LogicalTypes.md:
##########
@@ -684,44 +702,67 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
+should always be determined by the following rules if they cannot be determined
+as 3-level structures:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;

Review Comment:
   See my comment above.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1854468241


##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures

Review Comment:
   1-level isn't legacy (yet). It's an alternate way to specify lists without using the `LIST` annotation. These rules only apply for `LIST` annotated structures. So it's really just 1 & 2, but part of 1 is recognizing that a 3-level list with the repeated group named "array" or "<list_name>_tuple" is in fact a 2-level structure subject to the new Rule 4.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1859237857


##########
LogicalTypes.md:
##########
@@ -609,9 +609,20 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
-Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
-repeated fields, but not both. When using the annotations, no unannotated
-repeated types are allowed.
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}

Review Comment:
   I think this example is counter-productive. We don't want anyone using un-annotated lists and maps. While the paragraph above explains how to interpret un-annotated `repeated` fields, I don't want anyone to see an example here and think that it is something that should be copied. I think it is already clear enough and I would simply move on rather than drawing attention to this as a possibility.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1859970898


##########
LogicalTypes.md:
##########
@@ -609,9 +609,20 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
-Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
-repeated fields, but not both. When using the annotations, no unannotated
-repeated types are allowed.
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}

Review Comment:
   That make sense. Let me remove these examples first. I think a followup is to deprecate it by moving it to the backward compatibility section and adding strong words to discourage writers to emit it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1856842488


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   If naming does not satisfy Rule 4, then it is a 3-level structure without normal naming.
   
   ```
   // Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
   optional group my_list (LIST) {
     repeated group array {
       required binary str (STRING);
     };
   }
   
   // Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
   optional group my_list (LIST) {
     repeated group my_list_tuple {
       required binary str (STRING);
     };
   }
   
   // 3-level list: List<String> (nullable list, non-null elements)
   optional group my_list (LIST) {
     repeated group not_array_or_my_list_tuple {
       required binary str (STRING);
     };
   }
   ```
   
   (Sorry I have made an incorrect reply earlier)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1825643095


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   I think we'd better open another PR to discuss list types without annotation. It might be more complicated than the scope of this PR. I'm not sure if it should be allowed by the spec even though we are able to create the schema.
   
   EDIT: after thinking about it again, we may add following words to the end of the Lists section.
   
   > Some existing data does not even have the `LIST` annotation and simply uses
   > `repeated` repetition to annotate the element type. In this case, the element
   > type MUST be a primitive type and both the list and elements are required.
   > 
   > ```
   > // List<Integer> (non-null list, non-null elements)
   > repeated int32 num;
   >
   > // Struct<List<Integer>, List<String>> (non-null list, non-null elements)
   > optional group whatever {
   >   repeated int32 num;
   >   repeated binary str (STRING);
   > }
   > ```
   
   cc @tustvold @zeevm 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1826156685


##########
LogicalTypes.md:
##########
@@ -648,49 +648,66 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+Some existing data does not include the inner element layer, meaning that
+`LIST` annotates a 2-level structure. In contrast to 3-level structure, The
+repetition of the outer level `LIST`-annotated 2-level structure can be
+`optional`, `required`, or `required`. For backward-compatibility, the type of

Review Comment:
   Is this one too many `required`? Or do you mean to add `repeated` as well?



##########
LogicalTypes.md:
##########
@@ -648,49 +648,66 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+Some existing data does not include the inner element layer, meaning that
+`LIST` annotates a 2-level structure. In contrast to 3-level structure, The
+repetition of the outer level `LIST`-annotated 2-level structure can be
+`optional`, `required`, or `required`. For backward-compatibility, the type of
+elements in `LIST`-annotated structures should always be determined by the
+following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
-   elements are required.
+   elements are required. In this case, `LIST` annotates a 2-level structure.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, `LIST` annotates a
+   2-level structure and the element type is a Struct type with multiple fields.
+3. If the repeated field is a group (not `LIST`-annotated) with one `required`
+   or `optional` field, and is named either `array` or uses the `LIST`-annotated
+   group's name with `_tuple` appended, then the repeated type (a single-field
+   Struct type) is the element type and elements are required. This is a special
+   case of 3-level structure where the names are respected.
 4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+   field's repetition. In this case, `LIST` annotates a 2-level structure. Note
+   that the repeated field cannot be a 3-level LIST whose repetition must be

Review Comment:
   ```suggestion
      that the repeated field cannot be a 3-level LIST, as such a list's repetition must be
   ```



##########
LogicalTypes.md:
##########
@@ -648,49 +648,66 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+Some existing data does not include the inner element layer, meaning that
+`LIST` annotates a 2-level structure. In contrast to 3-level structure, The

Review Comment:
   ```suggestion
   `LIST` annotates a 2-level structure. In contrast to the 3-level structure, the
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1869146424


##########
LogicalTypes.md:
##########
@@ -684,44 +689,58 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
 should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated

Review Comment:
   Ok, thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1860313055


##########
LogicalTypes.md:
##########
@@ -609,9 +609,20 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
-Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
-repeated fields, but not both. When using the annotations, no unannotated
-repeated types are allowed.
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}
+```
+
+For all fields in the schema, implementations should use either `LIST` and

Review Comment:
   I think the original sentence is confusing and resulted in a relevant discussion: https://github.com/apache/parquet-format/pull/466#pullrequestreview-2432448521. Therefore I prefer to keep the current change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1823727500


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   > With a logical type annotation (I think only LIST and MAP apply here).
   
   After double check, I don't think MAP annotation will appear here because it always has three levels.
   
   > To point to the fact that for backwards compatibility it can also be repeated.
   
   The original statement is all about the formal three-level encoding. Adding `repeated` to it may further confuse readers. That's why I tried to clarify this in the backward compatibility section.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1839180500


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.

Review Comment:
   Yes, my intention is to differentiate the point 3 below which is a struct with a single field. I can remove the redundant statement if it is clear enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1844245348


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   > I'm not sure either why `parquet-rs` chosen to write "1-level" lists.
   
   It doesn't by default. It takes user intervention to pass a schema with 1-level lists to the record writer. The table writer will use the proper 3-level structure AFAICT.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841889654


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   I would suggest putting these examples, and the paragraph above, in a separate subsection, for example:
   ````markdown
   Implementations should use either `LIST` and `MAP` annotations _or_ unannotated
   repeated fields, but not both. When using the annotations, no unannotated
   repeated types are allowed.
   
   ### Unannotated repeated fields
   
   A repeated field that is neither contained by a `LIST`- or `MAP`-annotated
   group nor annotated by `LIST` or `MAP` should be interpreted as a
   required list of required elements where the element type is the type of
   the field.
   
   ```
   // List<Integer> (non-null list, non-null elements)
   repeated int32 num;
   
   // List<Tuple<Integer, String>> (non-null list, non-null elements)
   repeated group my_list {
     required int32 num;
     optional binary str (STRING);
   }
   ```
   
   It is heavily recommended that writers use either the LIST or MAP annotations,
   as outlined below, to make the intent clearer to readers.
   ````
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1842518889


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   I agree that it's slightly odd, but we would otherwise need a separate doc for interpretation of schemas as you say (and move any relevant contents out of `LogicalTypes.md`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1842715477


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {

Review Comment:
   If my comment above is correct, perhaps keep the Rule 5 example, but instead explain why it is not allowed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1839842596


##########
LogicalTypes.md:
##########
@@ -684,49 +686,99 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+
+// Rule 5: List<OneTuple<List<Integer>>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group foo {
+    repeated int32 bar;
+  };
+}
+```
+
+##### 1-level structure without `LIST` annotation
+
+Some existing data does not even have the `LIST` annotation and simply uses
+`repeated` repetition to annotate the element type. For backward-compatibility,
+both the list and elements are `required`.
+
+```
+// List<Integer> (non-null list, non-null elements)
+repeated int32 num;
+
+// Tuple<List<Integer>, List<String>> (non-null list, non-null elements)
+optional group my_list {
+  repeated int32 num;
+  repeated binary str (STRING);
+}
+
+// List<Tuple<Integer, String>> (non-null list, non-null elements)
+repeated group my_list {
+  required int32 num;
+  optional binary str (STRING);
+}
 ```

Review Comment:
   As @etseidl mentioned before, there is a [section](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md?plain=1#L604-L614) just before lists that clearly states that the repeated fields without annotation is supported. So, it is misleading that we explain the "1-level structure" at the backward compatibility section. Maybe move the examples to the existing section and keep only a reference that states the "1-level structure" is actually a 1st class citizen in the spec (and not deprecated).
   
   Or actually deprecate it now? Iceberg seems to only support the 3-level list. Meanwhile, there are engines still write the 1-level one, like parquet-rs. Even parquet-java writes these in case of the protobuf binding is used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2475684594

   > > Does it mean a file schema shall not contain both types of lists or that we shall not use annotated lists inside an unannotated one or vice-versa? @rdblue, WDYT?
   > 
   > I think this is addressed [here](https://github.com/apache/parquet-format/pull/17#discussion_r25807732). The intent was to not allow mixing list types within a schema, so I think all of the above.
   
   Thanks a lot, @etseidl. @wgtmac, maybe, we should articulate this to be 100% clear that at [this](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md?plain=1#L612-L614) part we are talking about the entire schema.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841819495


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;

Review Comment:
   (Seems it's due to legacy writer? Don't know how legacy writer write this)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1855135233


##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
+should always be determined by the following rules if they cannot be determined
+as 3-level structures:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field and the repetition of that
+   field is `repeated`, then its type is the element type and elements are

Review Comment:
   Changed the sentence a little bit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2502968117

   > The rules part is looking good, but I think that spending time documenting what people did incorrectly years ago makes the doc more confusing and increases chances that people will write invalid lists. I'd prefer to revert most of the changes that explain what people did incorrectly.
   
   I agree. But I think those can be posted on the pull-request description


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1856847518


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   But now the Rule 5 example is simply a 3-level list with different names. It would be `List<String>`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1855484507


##########
LogicalTypes.md:
##########
@@ -684,44 +702,67 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
+should always be determined by the following rules if they cannot be determined
+as 3-level structures:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`

Review Comment:
   So, the rule-4 doesn't means like (3), but inner-most element type is "required" ( not repeated ?)



##########
LogicalTypes.md:
##########
@@ -684,44 +702,67 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
+should always be determined by the following rules if they cannot be determined
+as 3-level structures:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;

Review Comment:
   Still cannot figure out rule3 and rule4 diffrences...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1856842488


##########
LogicalTypes.md:
##########
@@ -684,49 +697,76 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with a `repeated` field, then the repeated
+   field is the element type because the type cannot be a 3-level list.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated
    field's repetition.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Tuple<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<List<Integer>> (nullable outer list, non-null elements)
+optional group my_list (LIST) {
+  repeated group array (LIST) {
+    repeated int32 array;
+  };
+}
+
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 4: List<OneTuple<String>> (nullable list, non-null elements)

Review Comment:
   If naming does not satisfy Rule 4, then Rule 5 applies (though it produces the same schema).
   
   ```
   // Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
   optional group my_list (LIST) {
     repeated group array {
       required binary str (STRING);
     };
   }
   
   // Rule 4: List<OneTuple<String>> (nullable list, non-null elements)
   optional group my_list (LIST) {
     repeated group my_list_tuple {
       required binary str (STRING);
     };
   }
   
   // Rule 5: List<OneTuple<String>> (nullable list, non-null elements)
   optional group my_list (LIST) {
     repeated group not_array_or_my_list_tuple {
       required binary str (STRING);
     };
   }
   ```
   
   I think @rdblue have suggested not to change this rule for historical reason. Now it is pretty clear that these legacy namings do not make a difference, should we merge the last two rules to ignore their names?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2516027768

   @pitrou @gszadovszky @rdblue Do you have any concern with the latest change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1843105791


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   It looks strange to make it a subsection if the unannotated repeated fields are not deprecated. If we deprecate it, `Backward-compatibility rules` section can be organized as below:
   - 3-level structure with different field names
   - 2-level structure
   - Unannotated repeated fields
   
   It would be better to deprecate it in another change. Therefore I tend to keep it as is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1845468363


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   Thanks, @etseidl.
   
   Related to protobuf (the parquet-java binding for protobuf) this is the other way around currently. There is a flag for converting repeated protobuf elements to "3-level" lists but it is false by default, so it converts directly to similar repeated elements in the parquet schema ("1-level" list).
   Anyways it seems that neither parquet-rs nor parquet-java (protobuf) should have an issue creating proper "3-level" lists. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1827252641


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,

Review Comment:
   No, rule 3 is for a group with one `required` or `optional` field while rule 4 is for a `LIST`-annotated group with one `repeated` field. They do not overlap and the rule 3 implies that the group cannot have any annotation. So their orders do not matter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1824856711


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   > I think we should, but I am not clear where to add it since these lines are all talking about LIST-annotated structures.
   
   Perhaps around line 575 where the unannotated form is introduced. 🤷  I admit this is a bit of an awkward location, since this file is about logical types. Out of scope here, but perhaps a new markdown file that documents the various ways repetition is supported would be appropriate. I know it would help me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2479863682

   Apologies for muddying the waters, but I'm still trying to get this all clear in my head. I'm wondering if rather than adding a new rule, can we simply modify Rule 3 to say
   ```
   If the repeated field is a group with one field, is named either "array" or uses the LIST-annotated 
   group's name with "_tuple" appended, and would otherwise be a valid 3-level structure as outlined 
   above, then the repeated type is the element type and elements are required.
   ```
   The form that triggered all this discussion
   ```
   optional group a (LIST) {
     repeated group array (LIST) {
       repeated int32 array;
     }
   }
   ```
   is interpreted by some readers using Rule 3 because the repeated group is named `array`. If Rule 3 is modified as above, this example would not trigger Rule 3 due to a) the `repeated` repetition on the inner-most field, and b) the `LIST` annotation on the repeated group. Using Rule 4, the element type is the repeated field's type, which in this case is a Rule 1 primitive list, yielding a nested 2-level list (`List<List<Integer>>`) with non-nullable elements at both levels.
   
   Without the `LIST` annotation on the repeated group, the above becomes
   ```
   optional group a (LIST) {
     repeated group array {
       repeated int32 array;
     }
   }
   ```
   which is not valid at all. It cannot be a 3-level nor Rule 3 list due to the `repeated` inner element field. Rule 4 could be construed to say this is a `List<OneTuple<List<Integer>>`, but that would require mixing `LIST` annotated and unannotated lists, which has already been forbidden earlier in the specification.
   
   @rdblue objected to an earlier draft similar to what I'm proposing, but the new Rule 3 only works because of the `LIST` annotation on the repeated group, not because the innermost field is `repeated`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1835181742


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.

Review Comment:
   What is a "struct"? Do you mean the group with multiple fields? If so, then this is redundant isn't it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1841869882


##########
LogicalTypes.md:
##########
@@ -670,6 +681,8 @@ optional group array_of_arrays (LIST) {
 
 #### Backward-compatibility rules

Review Comment:
   The current spec (and it didn't change with this PR) specifies the "3-level" structure and the "unannotated" (aka. "1-level") list types as ones to be supported. So, according to the spec, a writer either use the first or the second one but cannot use both in the same schema.
   
   Since only the "3-level" structure is capable of properly describing nullability at any level, this should be the preferred one.
   
   We may start a discussion to deprecate the "1-level" list but I think this would be a different topic. This PR is about clarifying the backward compatibility rules only.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2484571721

   > Ok, so the new Rule 3 is in effect saying that if the child of the repeated middle group is repeated, then because 1-level and LIST/MAP annotated schemas cannot mix, the middle group cannot be the type so the child is the type. The side consequence of this is that the middle group has to have a LIST or MAP annotation to allow the 3rd level to be repeated
   
   No, the middle level can never be a LIST-annotated or MAP-annotated 3-level structure, because they cannot be `repeated`. As you have said, the middle level cannot be an unannotated group with a single repeated field due to mixing LIST-annotation and repeated field without annotation. So the only valid case for rule 3 is that the middle level is a LIST-annotated 2-level structure.
   
   > So is it true then that the repeated keyword may only be used for the child of a LIST or MAP annotated field when those annotations are in use? And if that's true, then a 2-level structure cannot be repeated as stated above?
   
   You're right. A LIST-annotated 2-level structure can only be repeated when it is a direct child of another LIST-annotated 2-level structure. I had missed the statement that disallows annotated and unannotated types to be mixed and modified these rules back and forth. Now this is pretty clear. Let me reflect these discussions in the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1843395569


##########
LogicalTypes.md:
##########
@@ -609,6 +609,17 @@ that is neither contained by a `LIST`- or `MAP`-annotated group nor annotated
 by `LIST` or `MAP` should be interpreted as a required list of required
 elements where the element type is the type of the field.
 
+```

Review Comment:
   To address the oddity of the proper placing of the "1-level" list and such, maybe, we should create a separate doc for nested types. Then, we can reference this doc from both Readme and the logical types doc for LIST and MAP (and MAP_KEY_VALUE). In this new doc we could even mention "struct"...
   
   I agree with @wgtmac. Let's discuss the deprecation in a separate PR, probably starting with a discussion thread on the dev list. It would impact `parquet-java` and `parquet-rs` at least. Not sure, if we can use the 3-level list for protobuf. I'm not an expert. I'm not sure either why `parquet-rs` chosen to write "1-level" lists. The guys there should also be involved in this discussion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1825643095


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   I think we'd better open another PR to discuss list types without annotation. It might be more complicated than the scope of this PR. I'm not sure if it should be allowed by the spec even though we are able to create the schema.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1839180500


##########
LogicalTypes.md:
##########
@@ -648,49 +650,105 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.

Review Comment:
   Yes, my intention is to differentiate the point 3 below which is a struct with a single field. I am remove the redundant statement if it is clear enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2472207420

   I have addressed the comments. Could you help review it again? @rdblue @gszadovszky 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #466:
URL: https://github.com/apache/parquet-format/pull/466#issuecomment-2492923161

   @gszadovszky @etseidl @rdblue @emkornfield Do you have any concern about this PR? I think we need to move forward now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1855134474


##########
LogicalTypes.md:
##########
@@ -684,44 +702,68 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures

Review Comment:
   I agree with @etseidl. This section only discusses LIST-annotated structures.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1869110188


##########
LogicalTypes.md:
##########
@@ -684,44 +689,58 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
+Some existing data does not include the inner element layer, resulting in a
+`LIST` that annotates a 2-level structure. Unlike the 3-level structure, the
+repetition of a 2-level structure can be `optional`, `required`, or `repeated`.
+When it is `repeated`, the `LIST`-annotated 2-level structure can only serve as
+an element within another `LIST`-annotated 2-level structure.
+
+For backward-compatibility, the type of elements in `LIST`-annotated structures
 should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
    element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
+3. If the repeated field is a group with one field with `repeated` repetition,
+   then its type is the element type and elements are required.
+4. If the repeated field is a group with one field and is named either `array`
    or uses the `LIST`-annotated group's name with `_tuple` appended then the
    repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
+5. Otherwise, the repeated field's type is the element type with the repeated

Review Comment:
   Ok, I added one with a slightly different schema. The inner field is `optional` in the new example. Please take a look.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1821868722


##########
LogicalTypes.md:
##########
@@ -654,13 +654,18 @@ should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+2. If the repeated field is a group (without `LIST` or `MAP` annotation) with
+   multiple fields, then its type (i.e. a multi-field Tuple) is the element
+   type and elements are required.
+3. If the repeated field is a group (without `LIST` or `MAP` annotation) with
+   one field (with `required` or `optional` repetition) and is named either
+   `array` or uses the `LIST`-annotated group's name with `_tuple` appended,
+   then the repeated type is the element type and elements are required. Note
+   that the element type resolves to a single-field Tuple type.
 4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+   field's repetition and elements are required. Note that the element type
+   resolves to List or Map type if the repeated field is `LIST`-annotated or
+   `MAP`-annotated; otherwise the element type resolves to a Tuple type.

Review Comment:
   The example below uses `Tuple` and `OneTuple` and we'd better be consistent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1827155710


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,

Review Comment:
   with only one?



##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.

Review Comment:
   A unrelated: I just find this "elements are required" is a bit tricky for me...



##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,
+   then the element type is a list type with 2-level structure and elements are
+   required.
+5. Otherwise, the repeated field's type is the element type with the repeated
+   field's repetition. Please note that the repeated field here (a group with
+   one field) cannot be `LIST`-annotated or `MAP`-annotated 3-level structure,
+   as such a group's repetition must be `required` or `optional`.
 
 Examples that can be interpreted using these rules:
 
 ```
-// List<Integer> (nullable list, non-null elements)
+// Rule 1: List<Integer> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated int32 element;
 }
 
-// List<Tuple<String, Integer>> (nullable list, non-null elements)
+// Rule 2: List<Struct<String, Integer>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group element {
     required binary str (STRING);
     required int32 num;
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group array {
     required binary str (STRING);
   };
 }
 
-// List<OneTuple<String>> (nullable list, non-null elements)
+// Rule 3: List<Struct<String>> (nullable list, non-null elements)
 optional group my_list (LIST) {
   repeated group my_list_tuple {
     required binary str (STRING);
   };
 }
+

Review Comment:
   do we have sample for Rule 5?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1823719158


##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   I doesn't hurt to make it explicit.
   
   > I'm also wondering if we should add an example above for the non-LIST annotated repeating fields, or should that be a new PR?
   
   I think we should, but I am not clear where to add it since these lines are all talking about LIST-annotated structures.



##########
LogicalTypes.md:
##########
@@ -655,10 +655,12 @@ should always be determined by the following rules:
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
 2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+   element type and elements are required. In this case, the element type is
+   a Struct type with multiple fields.
+3. If the repeated field is a group (without annotation) with one `required` or

Review Comment:
   It doesn't hurt to make it explicit.
   
   > I'm also wondering if we should add an example above for the non-LIST annotated repeating fields, or should that be a new PR?
   
   I think we should, but I am not clear where to add it since these lines are all talking about LIST-annotated structures.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1827252641


##########
LogicalTypes.md:
##########
@@ -648,49 +650,92 @@ optional group my_list (LIST) {
 }
 ```
 
-Some existing data does not include the inner element layer. For
-backward-compatibility, the type of elements in `LIST`-annotated structures
-should always be determined by the following rules:
+##### 2-level structure
+
+Some existing data does not include the inner element layer, meaning that `LIST`
+annotates a 2-level structure. In contrast to 3-level structure, the repetition
+of 2-level structure can be `optional`, `required`, or `repeated`.
+
+```
+<list-repetition> group <name> (LIST) {
+  repeated <element-type> <element-name>;
+}
+```
+
+For backward-compatibility, the type of elements in `LIST`-annotated 2-level
+structures should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
-4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+2. If the repeated field is a group with multiple fields, then its type (Struct
+   type with multiple fields) is the element type and elements are required.
+3. If the repeated field is a group with one `required` or `optional` field,
+   and is named either `array` or uses the `LIST`-annotated group's name with
+   `_tuple` appended, then the repeated type (Struct type with single field) is
+   the element type and elements are required.
+4. If the repeated field is a `LIST`-annotated group with one `repeated` field,

Review Comment:
   No, rule 3 is for a group with one `required` or `optional` field while rule 4 is for a `LIST`-annotated group with one `repeated` field. They do not overlap and the rule 3 implies that the group cannot have any annotation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1821811276


##########
LogicalTypes.md:
##########
@@ -654,13 +654,18 @@ should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
-   element type and elements are required.
-3. If the repeated field is a group with one field and is named either `array`
-   or uses the `LIST`-annotated group's name with `_tuple` appended then the
-   repeated type is the element type and elements are required.
+2. If the repeated field is a group (without `LIST` or `MAP` annotation) with
+   multiple fields, then its type (i.e. a multi-field Tuple) is the element
+   type and elements are required.
+3. If the repeated field is a group (without `LIST` or `MAP` annotation) with
+   one field (with `required` or `optional` repetition) and is named either
+   `array` or uses the `LIST`-annotated group's name with `_tuple` appended,
+   then the repeated type is the element type and elements are required. Note
+   that the element type resolves to a single-field Tuple type.
 4. Otherwise, the repeated field's type is the element type with the repeated
-   field's repetition.
+   field's repetition and elements are required. Note that the element type
+   resolves to List or Map type if the repeated field is `LIST`-annotated or
+   `MAP`-annotated; otherwise the element type resolves to a Tuple type.

Review Comment:
   I don't know the worlding of "Tuple type" is better or "Struct type" is better...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-465: Clarify backward-compatibility rules on LIST type [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #466:
URL: https://github.com/apache/parquet-format/pull/466#discussion_r1822227723


##########
LogicalTypes.md:
##########
@@ -654,12 +654,15 @@ should always be determined by the following rules:
 
 1. If the repeated field is not a group, then its type is the element type and
    elements are required.
-2. If the repeated field is a group with multiple fields, then its type is the
+2. If the repeated field is a group with `LIST` or `MAP` annotation, then the

Review Comment:
   ```
   1. If the repeated field is not a group, then its type is the element type and
      elements are required.
   2. If the repeated field is a group with multiple fields, then its type is the
      element type and elements are required. To be clear, if the group does not
      have annotation, the element type resolves to a multi-field Tuple. If the
      group is `LIST`-annotated or `MAP`-annotated, it should resolve to List or
      Map type, respectively.
   3. If the repeated field is a group (without annotation) with one `required` or
      `optional` field, and is named either `array` or uses the `LIST`-annotated
      group's name with `_tuple` appended, then the repeated type (a single-field
      Tuple type) is the element type and elements are required.
   4. Otherwise, the repeated field's type is the element type with the repeated
      field's repetition.
   ```
   
   This is another version I've come up with, which implicitly applies `repeated field is a group with LIST or MAP annotation` to the last rule `Otherwise, the repeated field's type is the element type with the repeated field's repetition.`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org