You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "HuaHuaY (via GitHub)" <gi...@apache.org> on 2026/06/27 07:53:26 UTC

[PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

HuaHuaY opened a new pull request, #591:
URL: https://github.com/apache/parquet-format/pull/591

   ### Rationale for this change
   
   Allow VARIANT value to be omitted.
   
   There are some cases in `parquet-testing` https://github.com/apache/parquet-testing/tree/master/shredded_variant which don't have `value` field. 
   
   ```sh
   # Doesn't have top value field.
   parquet-testing/shredded_variant on  master 
   ❯ ~/arrow/cpp/out/build/ninja-debug/debug/parquet-dump-schema case-131.parquet 
   required group field_id=-1 table {
     required int32 field_id=1 id;
     required group field_id=2 var (Variant(1)) {
       required binary field_id=-1 metadata;
       optional int32 field_id=-1 typed_value;
     }
   }
   
   # Doesn't have inner value field.
   parquet-testing/shredded_variant on  master 
   ❯ ~/arrow/cpp/out/build/ninja-debug/debug/parquet-dump-schema case-132.parquet
   required group field_id=-1 table {
     required int32 field_id=1 id;
     optional group field_id=2 var (Variant(1)) {
       required binary field_id=-1 metadata;
       optional binary field_id=-1 value;
       optional group field_id=-1 typed_value {
         required group field_id=-1 a {
           optional int32 field_id=-1 typed_value;
         }
         required group field_id=-1 b {
           optional binary field_id=-1 typed_value (String);
         }
       }
     }
   }
   ```
   
   ### What changes are included in this PR?
   
   Updated the descriptions in the variant-related Markdown files.
   
   ### Do these changes have PoC implementations?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4918610771

   Specifcally, if case-132 is actually invalid, perhaps we should rename https://github.com/apache/parquet-testing/blob/master/shredded_variant/case-132_row-0.variant.bin to case-132-INVALID_row-0.variant.bin 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4907983005

   If arrow-rs is not producing the `value` field, then it is out of spec. But that's also what this test case is for, to ensure that we can read things that are out of spec. I don't think we want to add this as a "bad" example, if what you mean is to test that readers fail. It is reasonable to read the data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4939397357

   To add some clarity following the arrow-rs report (apache/arrow-rs#10306), here is a survey of how implementations currently handle a variant group that omits the `value` column:
   
   | implementation | reading a value-less group | writing |
   |---|---|---|
   | DuckDB | **rejects**: ["The Variant column must have 'metadata' and 'value' as the first two columns"](https://github.com/duckdb/duckdb/blob/985b2f2515adf44ea65dd4239c7bfb30981b94dc/extension/parquet/reader/variant_column_reader.cpp#L54-L62) | always emits `value` ([writer](https://github.com/duckdb/duckdb/blob/985b2f2515adf44ea65dd4239c7bfb30981b94dc/extension/parquet/writer/variant/convert_variant.cpp)) |
   | Spark | reads ([guards every `value` access](https://github.com/apache/spark/blob/c470f3dce544ad46ffb748949b6476d3cbc8d443/common/variant/src/main/java/org/apache/spark/types/variant/ShreddingUtils.java#L139-L168)) | always emits `value` ([`variantShreddingSchema` includes it at every level](https://github.com/apache/spark/blob/c470f3dce544ad46ffb748949b6476d3cbc8d443/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala#L485-L512)) |
   | parquet-java | reads ([`containsField("value")` guards](https://github.com/apache/parquet-java/blob/ea349cd3e97538574604ee15248713f0c35dd0d9/parquet-variant/src/main/java/org/apache/parquet/variant/VariantConverters.java#L151)) | assumes `value` exists whenever a row needs the binary fallback ([`getFieldIndex` throws otherwise](https://github.com/apache/parquet-java/blob/ea349cd3e97538574604ee15248713f0c35dd0d9/parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java#L99-L108)) |
   | Arrow C++ | no shredding support yet; its unshredded storage validation [requires non-nullable `metadata` and `value`](https://github.com/apache/arrow/blob/5815dc0b4e3d9239828b47d427c9759fbc506e09/cpp/src/arrow/extension/parquet_variant.cc#L72-L105) | n/a |
   | arrow-go | reads; type validation currently requires only ["at least one of `value` or `typed_value`"](https://github.com/apache/arrow-go/blob/68e738f86506f61796e57d68551269552d76cdbf/arrow/extensions/variant.go#L176-L181) | may omit |
   | arrow-rs | currently reads (validation gap); fix in progress to reject (apache/arrow-rs#10306) | currently may omit `value`; fix in progress to always emit it (apache/arrow-rs#10306) |
   | cuDF | reads the raw struct without reconstructing values; [its tests assert the `value` child is present even for shredded columns](https://github.com/rapidsai/cudf/blob/0f98463d28c8346b2736c31420e4ef414228f5e2/python/cudf/cudf/tests/input_output/test_parquet_variant.py) (fixtures are DuckDB-written) | n/a |
   
   This supports the spec text as it stands: the implementations that validate variant group structure (DuckDB, Arrow C++) require `value`, arrow-rs is adopting the same, and every surveyed writer that produces spec-targeted files emits it. Where value-less groups are readable, it is by absence of validation rather than considered support.
   
   On how the divergence propagated: the `shredded_variant` cases in parquet-testing (generated from the Iceberg test suite in apache/parquet-testing#91) include four cases that omit `value` (41, 131, 132, 138) labeled as *valid* cases with expected outputs rather than with the `-INVALID` convention. arrow-rs and arrow-go both run that corpus in CI, so the lenient interpretation looked conformance-verified to both. I have filed https://github.com/apache/parquet-testing/issues/116 proposing those cases be relabeled per @alamb's suggestion above.
   
   With the spec question settled, the remaining work is enforcement in implementations: arrow-rs's is in progress under apache/arrow-rs#10306, and arrow-go's type validation currently accepts the omission and would need a similar update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4841123538

   Could you help review this? @aihuaxu @steveloughran 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4859939343

   I don't think this is correct.
   
   The test cases are the ones that we use in Parquet Java and Iceberg to ensure that we can read variant data as long as it can be interpreted correctly. Being liberal about what we accept does not mean that it is a good idea for writers to produce values that way. We don't need to adjust the spec, this is just a reasonable choice for readers so they don't needlessly fail.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4847717282

   Perhaps @rdblue  or @gene-db (the authors of the original spec in https://github.com/apache/parquet-format/pull/456 https://github.com/apache/parquet-format/pull/460) can help take a look
   
   The example files were generated by spark so I assume they are now out in the wild and we shoudl adjust the spec accordingly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4918600029

   I filed a ticket in arrow-rs to track this report
   - https://github.com/apache/arrow-rs/issues/10306
   
   I think we tried to make the malformed variant examples in the parquet-testing corpus clear (aka what is in https://github.com/apache/parquet-testing/tree/master/shredded_variant)  by clearly labeling them as INVALID
   
   e.g. https://github.com/apache/parquet-testing/blob/master/shredded_variant/case-043-INVALID.parquet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4981106862

   @sdf-jkl  made a proposal to mark the cases with missing value columns as INVALID in the parquet-testing repo:
   - https://github.com/apache/parquet-testing/pull/117


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4874566287

   @zeroshade I think so.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "steveloughran (via GitHub)" <gi...@apache.org>.
steveloughran commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4906394849

   if it's not optional, then arrow-rs must stop issuing shredded variant fields without it; a variant without the value should be added to the test file repository as a bad data example.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4816035024

   @alamb @emkornfield @zeroshade Please take a look. I found that arrow-rs already allows to omit value array. https://github.com/apache/arrow-rs/blob/44f377284e332925ade540d916f52c04a44f8484/parquet-variant-compute/src/variant_array.rs#L863 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-5065830876

   Because https://github.com/apache/parquet-testing/pull/117 has been merged, I close this pull request.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4897543546

   @zeroshade and @wgtmac, the wording in the current spec is clear:
   
   > The group must contain a field named `metadata` and a field named `value`.
   
   This spec has been adopted by the community and has deployed implementations, including proprietary implementations.
   
   The change proposed here is a compatibility breaking change because it would allow a field that is currently required to be omitted. We should not make changes like this to the spec because it could break implementations that rely on this field. A breaking change like this would need to go into a new compatibility version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #591:
URL: https://github.com/apache/parquet-format/pull/591#discussion_r3493077552


##########
VariantEncoding.md:
##########
@@ -42,13 +42,14 @@ The [Variant Shredding specification](VariantShredding.md) describes the details
 
 ## Variant in Parquet
 
-A Variant value in Parquet is represented by a group with 2 fields, named `value` and `metadata`.
+An unshredded Variant value in Parquet is represented by a group with 2 fields, named `value` and `metadata`.
+A shredded Variant value adds a `typed_value` field, and may omit `value` when all values can be represented by `typed_value`.
 
 * The Variant group must be annotated with the `VARIANT` logical type.
-* Both fields `value` and `metadata` must be of type `binary` (called `BYTE_ARRAY` in the Parquet thrift definition).
+* The `metadata` field must be of type `binary` (called `BYTE_ARRAY` in the Parquet thrift definition). When present, the `value` field must also be of type `binary`.
 * The `metadata` field is `required` and must be a valid Variant metadata, as defined below.
-* The `value` field must be annotated as `required` for unshredded Variant values, or `optional` if parts of the value are [shredded](VariantShredding.md) as typed Parquet columns.
-* When present, the `value` field must be a valid Variant value, as defined below. 
+* The `value` field must be annotated as `required` for unshredded Variant values, or omitted or annotated as `optional` if the Variant value is [shredded](VariantShredding.md) as typed Parquet columns.

Review Comment:
   not a fan of the wording of "or omitted or annotated" maybe something like:
   
   > The `value` field must be annotated as `required` for unshredded Variant values. If the Variant value is [shredded](VariantShredding.md) as typed Parquet columns, then then `value` field may be annotated as `optional` or omitted entirely if ALL values can be represented in the typed column.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #591:
URL: https://github.com/apache/parquet-format/pull/591#issuecomment-4861583238

   Thanks @rdblue for the explanation! Then I agree that this spec doesn't need any change. Perhaps we have two follow-ups to avoid future confusion:
   - Fix wording in https://github.com/apache/arrow/blob/f10c93c000775c5fb30ca230f540358611091589/docs/source/format/CanonicalExtensions.rst?plain=1#L448 to align with the Parquet spec. cc @zeroshade @alamb 
   - Clarify the purpose of these test files in https://github.com/apache/parquet-testing/blob/master/shredded_variant/README.md
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-590: Allow VARIANT value to be omitted [parquet-format]

Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY closed pull request #591: GH-590: Allow VARIANT value to be omitted
URL: https://github.com/apache/parquet-format/pull/591


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org