You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2024/05/28 22:07:32 UTC

[PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

etseidl opened a new pull request, #251:
URL: https://github.com/apache/parquet-format/pull/251

   Changes instances of 'binary' to BYTE_ARRAY where appropriate. Also fixes some uses of FIXED_LEN_BYTE_ARRAY.
   
   Make sure you have checked _all_ steps below.
   
   ### Jira
   
   - [x] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR"
     - https://issues.apache.org/jira/browse/PARQUET-2485
     - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   
   ### Commits
   
   - [x] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Documentation
   
   - [ ] In case of new functionality, my PR adds documentation that describes how to use it.
     - All the public functions and the classes in the PR contain Javadoc that explain what it does
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #251:
URL: https://github.com/apache/parquet-format/pull/251#issuecomment-2136187544

   Note: I've left 'binary' in the schema examples for now since I'm not sure if the current parquet-cli still uses 'binary' when printing file schemas.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #251:
URL: https://github.com/apache/parquet-format/pull/251#issuecomment-2137766579

   > > Note: I've left 'binary' in the schema examples for now since I'm not sure if the current parquet-cli still uses 'binary' when printing file schemas.
   > 
   > Yes, I think we need to fix this as well.
   
   I verified that parquet-cli 1.14.0 still uses 'binary' for BYTE_ARRAY. I welcome suggestions for the schema examples in `LogicalTypes.md`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1635669939


##########
LogicalTypes.md:
##########
@@ -59,7 +60,7 @@ Compatibility considerations are mentioned for each annotation in the correspond
 
 ### STRING
 
-`STRING` may only be used to annotate the binary primitive type and indicates
+`STRING` may only be used to annotate the BYTE_ARRAY primitive type and indicates

Review Comment:
   It seems the spec is unclear about whether or not `STRING` and `ENUM` can annotate `FIXED_LENGTH_BYTE_ARRAY`. Literally it is reasonable to annotate `FIXED_LENGTH_BYTE_ARRAY`, right? I'm not sure if there is any use case in the wild.



##########
src/main/thrift/parquet.thrift:
##########
@@ -151,14 +151,14 @@ enum ConvertedType {
   /**
    * An embedded JSON document
    *
-   * A JSON document embedded within a single UTF8 column.
+   * A JSON document embedded within a single BYTE_ARRAY(STRING) column.

Review Comment:
   Should we keep it as is? This is the deprecated `ConvertedType` section where `UTF8` is used for string type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1635792297


##########
src/main/thrift/parquet.thrift:
##########
@@ -151,14 +151,14 @@ enum ConvertedType {
   /**
    * An embedded JSON document
    *
-   * A JSON document embedded within a single UTF8 column.
+   * A JSON document embedded within a single BYTE_ARRAY(STRING) column.

Review Comment:
   Oh, good point. I'll revert.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1643253963


##########
LogicalTypes.md:
##########
@@ -228,7 +229,7 @@ integer. A precision too large for the underlying type (see below) is an error.
   warning
 * `fixed_len_byte_array`: precision is limited by the array size. Length `n`
   can store &lt;= `floor(log_10(2^(8*n - 1) - 1))` base-10 digits
-* `binary`: `precision` is not limited, but is required. The minimum number of
+* `byte_array`: `precision` is not limited, but is required. The minimum number of

Review Comment:
   Yes, the formatting here is odd...all of the type names are lower case. Also, the use of back ticks for type names is inconsistent.  Perhaps (as you suggest) for this PR we can solely worry about binary->byte_array, and then do a second pass to fix capitalization and quoting.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1643280414


##########
LogicalTypes.md:
##########
@@ -228,7 +229,7 @@ integer. A precision too large for the underlying type (see below) is an error.
   warning
 * `fixed_len_byte_array`: precision is limited by the array size. Length `n`
   can store &lt;= `floor(log_10(2^(8*n - 1) - 1))` base-10 digits
-* `binary`: `precision` is not limited, but is required. The minimum number of
+* `byte_array`: `precision` is not limited, but is required. The minimum number of

Review Comment:
   Looking at this more, it seems I'm introducing more inconsistency with the quoting. I'll try to clean that up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1644865653


##########
LogicalTypes.md:
##########
@@ -59,7 +60,7 @@ Compatibility considerations are mentioned for each annotation in the correspond
 
 ### STRING
 
-`STRING` may only be used to annotate the binary primitive type and indicates
+`STRING` may only be used to annotate the BYTE_ARRAY primitive type and indicates

Review Comment:
   I agree with this assesment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #251:
URL: https://github.com/apache/parquet-format/pull/251#issuecomment-2136353533

   > Note: I've left 'binary' in the schema examples for now since I'm not sure if the current parquet-cli still uses 'binary' when printing file schemas.
   
   Yes, I think we need to fix this as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #251:
URL: https://github.com/apache/parquet-format/pull/251#issuecomment-2167078970

   LGTM cc @pitrou @alamb @tustvold 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1643208427


##########
LogicalTypes.md:
##########
@@ -554,8 +555,8 @@ The sort order used for `JSON` is unsigned byte-wise comparison.
 
 ### BSON
 
-`BSON` is used for an embedded BSON document. It must annotate a `binary`
-primitive type. The `binary` data is interpreted as an encoded BSON document as
+`BSON` is used for an embedded BSON document. It must annotate a BYTE_ARRAY
+primitive type. The byte array data is interpreted as an encoded BSON document as

Review Comment:
   ```suggestion
   primitive type. The BYTE_ARRAY data is interpreted as an encoded BSON document as
   ```



##########
LogicalTypes.md:
##########
@@ -228,7 +229,7 @@ integer. A precision too large for the underlying type (see below) is an error.
   warning
 * `fixed_len_byte_array`: precision is limited by the array size. Length `n`
   can store &lt;= `floor(log_10(2^(8*n - 1) - 1))` base-10 digits
-* `binary`: `precision` is not limited, but is required. The minimum number of
+* `byte_array`: `precision` is not limited, but is required. The minimum number of

Review Comment:
   Perhaps we can change this to name the types using the same names as elswhere:
   
   ```suggestion
   * `FIXED_LEN_BYTE_ARRAY `: precision is limited by the array size. Length `n`
     can store &lt;= `floor(log_10(2^(8*n - 1) - 1))` base-10 digits
   * `BYTE_ARRAY `: `precision` is not limited, but is required. The minimum number of



##########
LogicalTypes.md:
##########
@@ -211,8 +212,8 @@ unsigned integers with 8, 16, 32, or 64 bit width.
 `DECIMAL` annotation represents arbitrary-precision signed decimal numbers of
 the form `unscaledValue * 10^(-scale)`.
 
-The primitive type stores an unscaled integer value. For byte arrays, binary
-and fixed, the unscaled number must be encoded as two's complement using
+The primitive type stores an unscaled integer value. For byte arrays, variable
+and fixed-length, the unscaled number must be encoded as two's complement using

Review Comment:
   We can probably make this wording clearer by just using the names directly: 
   
   ```suggestion
   The primitive type stores an unscaled integer value. For BYTE_ARRAY and 
   FIXED_LEN_BYTE_ARRAY, the unscaled number must be encoded as two's complement using
   ```



##########
LogicalTypes.md:
##########
@@ -544,8 +545,8 @@ Embedded types do not have type-specific orderings.
 
 ### JSON
 
-`JSON` is used for an embedded JSON document. It must annotate a `binary`
-primitive type. The `binary` data is interpreted as a UTF-8 encoded character
+`JSON` is used for an embedded JSON document. It must annotate a BYTE_ARRAY
+primitive type. The byte array data is interpreted as a UTF-8 encoded character

Review Comment:
   ```suggestion
   primitive type. The BYTE_ARRAY data is interpreted as a UTF-8 encoded character
   ```



##########
LogicalTypes.md:
##########
@@ -59,7 +60,7 @@ Compatibility considerations are mentioned for each annotation in the correspond
 
 ### STRING
 
-`STRING` may only be used to annotate the binary primitive type and indicates
+`STRING` may only be used to annotate the BYTE_ARRAY primitive type and indicates

Review Comment:
   I think the existing wording is pretty clear of we take "binary primitive type" to mean BYTE_ARRAY and thus in my opinion this is not a change to the spec. 
   
   Perhaps we could send a note to dev@parquet.apache.org just highlighting this clarification in case someone wants to chime in and say they read it to mean FIXED_LENGTH_BYTE_ARRAY was also supported



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #251:
URL: https://github.com/apache/parquet-format/pull/251


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1635788871


##########
LogicalTypes.md:
##########
@@ -59,7 +60,7 @@ Compatibility considerations are mentioned for each annotation in the correspond
 
 ### STRING
 
-`STRING` may only be used to annotate the binary primitive type and indicates
+`STRING` may only be used to annotate the BYTE_ARRAY primitive type and indicates

Review Comment:
   I don't know. For `DECIMAL` the spec calls out "byte arrays, binary and fixed" as valid physical types. I'd take the lack of mention of fixed-length here to indicated that only the `BYTE_ARRAY` physical type is allowed. Do any current implementations allow fixed-length strings?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #251:
URL: https://github.com/apache/parquet-format/pull/251#issuecomment-2136360316

   I just noticed I left in the converted type UTF8 rather than using the proper logical type name STRING. I'll fix that up tomorrow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1644861464


##########
LogicalTypes.md:
##########
@@ -59,7 +60,7 @@ Compatibility considerations are mentioned for each annotation in the correspond
 
 ### STRING
 
-`STRING` may only be used to annotate the binary primitive type and indicates
+`STRING` may only be used to annotate the BYTE_ARRAY primitive type and indicates

Review Comment:
   The sense I'm getting from the M/L is that people are open to adding it, but haven't seen any evidence that any one actually supports adding `STRING` to `FLBA`. IMO we should ship this as is, and start a new thread to gauge support for actually modifying the spec to allow for fixed length strings. Given the variable width of UTF-8 characters, I'd think padding would have to be added to account for up to 4 bytes per character.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-2485: Be more consistent with BYTE_ARRAY types [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #251:
URL: https://github.com/apache/parquet-format/pull/251#discussion_r1645266888


##########
LogicalTypes.md:
##########
@@ -59,7 +60,7 @@ Compatibility considerations are mentioned for each annotation in the correspond
 
 ### STRING
 
-`STRING` may only be used to annotate the binary primitive type and indicates
+`STRING` may only be used to annotate the BYTE_ARRAY primitive type and indicates

Review Comment:
   I agree that we'd better keep it as is for now. Let me merge this. We can always change the content when there is a consensus.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org