[DISCUSS] Allow VARIANT value field to be omitted

[Zehua Zou <ze...@gmail.com>]

Hi all,

I found that parquet-format doesn’t allow VARIANT field value to be omitted, not like Arrow.

Arrow’s documents say "At least one (or both) of the following… A field named value… A field named typed_value…”. https://arrow.apache.org/docs/format/CanonicalExtensions.html#parquet-variant

But Parquet’s documents just say "The `value` field must be annotated as `required` for unshredded Variant values, or `optional` if parts of the value are [shredded](VariantShredding.md) as typed Parquet columns.”. https://github.com/apache/parquet-format/blob/master/VariantEncoding.md

But there are some cases in parquet-testing don’t follow this behavior. For examples,

parquet-testing/shredded_variant/case-131.parquet:

required group field_id=-1 table {
  required int32 field_id=1 id;
  required group field_id=2 var (Variant(1)) {
    required binary field_id=-1 metadata;
    optional int32 field_id=-1 typed_value;
  }
}

parquet-testing/shredded_variant/case-132.parquet:

required group field_id=-1 table {
  required int32 field_id=1 id;
  optional group field_id=2 var (Variant(1)) {
    required binary field_id=-1 metadata;
    optional binary field_id=-1 value;
    optional group field_id=-1 typed_value {
      required group field_id=-1 a {
        optional int32 field_id=-1 typed_value;
      }
      required group field_id=-1 b {
        optional binary field_id=-1 typed_value (String);
      }
    }
  }
}

I created an issue which records more cases. https://github.com/apache/parquet-format/issues/590

Should we modify parquet-format’s description and allow value field to be omitted? And I found arrow-rs has already allowed value field to be omitted. https://github.com/apache/arrow-rs/blob/7616e10f12894232eacb807993e55427a067d061/parquet-variant-compute/src/variant_array.rs#L858

I have submitted a PR to adjust the description. https://github.com/apache/parquet-format/pull/591

Best,
Zehua Zou