You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "cashmand (via GitHub)" <gi...@apache.org> on 2025/05/20 21:46:50 UTC

[PR] Implement Variant parquet writer [parquet-java]

cashmand opened a new pull request, #3221:
URL: https://github.com/apache/parquet-java/pull/3221

   <!--
   Thanks for opening a pull request!
   
   If you're new to Parquet-Java, information on how to contribute can be found here: https://parquet.apache.org/docs/contribution-guidelines/contributing
   
   Please open a GitHub issue for this pull request: https://github.com/apache/parquet-java/issues/new/choose
   and format pull request title as below:
   
       GH-${GITHUB_ISSUE_ID}: ${SUMMARY}
   
   or simply use the title below if it is a minor issue:
   
       MINOR: ${SUMMARY}
   
   -->
   
   ### Rationale for this change
   
   
   ### What changes are included in this PR?
   
   
   ### Are these changes tested?
   
   
   ### Are there any user-facing changes?
   
   
   <!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the actual Github issue id. -->
   <!-- Closes #${GITHUB_ISSUE_ID} -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155714340


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());

Review Comment:
   Sounds reasonable to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155720733


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;

Review Comment:
   This can be final.
   
   Also, I like to keep final instance fields located together before others.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099016302


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)

Review Comment:
   Should this be `UNSHREDDED_SCHEMA` or `UNSHREDDED_GROUP`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130599035


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -848,4 +848,73 @@ static HashMap<String, Integer> getMetadataMap(ByteBuffer metadata) {
     }
     return result;
   }
+
+  /**
+   * Computes the actual size (in bytes) of the Variant value at `value[pos...]`

Review Comment:
   Is this still relevant? It looks like it was copied from an instance method that expected `value` and `pos` fields.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130727298


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;

Review Comment:
   Please remove wildcard imports.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130618861


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -848,4 +848,73 @@ static HashMap<String, Integer> getMetadataMap(ByteBuffer metadata) {
     }
     return result;
   }
+
+  /**
+   * Computes the actual size (in bytes) of the Variant value at `value[pos...]`
+   * @param value The Variant value
+   * @return The size (in bytes) of the Variant value
+   */
+  public static int valueSize(ByteBuffer value) {
+    int pos = value.position();
+    checkIndex(pos, value.limit());
+    int basicType = value.get(pos) & BASIC_TYPE_MASK;
+    int typeInfo = (value.get(pos) >> BASIC_TYPE_BITS) & PRIMITIVE_TYPE_MASK;
+    switch (basicType) {
+      case SHORT_STR:
+        return 1 + typeInfo;

Review Comment:
   This made me go check the spec. It may be easier to maintain/read if you just duplicated the logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155717992


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  private static final String LIST_REPEATED_NAME = "list";
+  private static final String LIST_ELEMENT_NAME = "element";
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField, value)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  // Return true if the logical type is a decimal with the same scale as the provided value, with enough
+  // precision to hold the value. The provided value must be a decimal.
+  private boolean compatibleDecimalType(Variant value, LogicalTypeAnnotation logicalType) {

Review Comment:
   This can be static, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099013816


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static MessageType readSchema = parquetSchema(variantGroup);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(variantGroup);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(readSchema);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataRawBytes();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataRawBytes();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataRawBytes());
+    vRecord.put(1, v.getValueRawBytes());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(readSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      // assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+      if (!((GenericRecord) record.get(1)).get(1).equals(((GenericRecord) actual.get(1)).get(1))) {
+        assertTrue(false);
+      }
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant v : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();
+      for (int i = 0; i < VARIANTS.length; i++) {
+        expected.add(createRecord(i, VARIANTS[i]));
+      }
+
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      List<GenericRecord> actual = writeAndRead(testSchema, expected);
+      // TODO: CHECK RESULTS
+    }
+  }
+
+  // Write schema contains the full shredding schema. Read schema should just be a value/metadata pair.
+  private static class TestSchema {
+    MessageType writeSchema;
+    MessageType readSchema;
+
+    TestSchema(MessageType writeSchema, MessageType readSchema) {
+      this.writeSchema = writeSchema;
+      this.readSchema = readSchema;
+    }
+  }
+
+  /**
+   * This is a custom Parquet writer builder that injects a specific Parquet schema and then uses
+   * the Avro object model. This ensures that the Parquet file's schema is exactly what was passed.
+   */
+  private static class TestWriterBuilder extends ParquetWriter.Builder<GenericRecord, TestWriterBuilder> {
+    private MessageType schema = null;
+
+    protected TestWriterBuilder(Path path) {
+      super(path);
+    }
+
+    TestWriterBuilder withFileType(MessageType schema) {
+      this.schema = schema;
+      return self();
+    }
+
+    @Override
+    protected TestWriterBuilder self() {
+      return this;
+    }
+
+    @Override
+    protected WriteSupport<GenericRecord> getWriteSupport(Configuration conf) {
+      return new AvroWriteSupport<>(schema, new AvroSchemaConverter().convert(schema), GenericData.get());
+    }
+  }
+
+  GenericRecord writeAndRead(TestSchema testSchema, GenericRecord record) throws IOException {
+    List<GenericRecord> result = writeAndRead(testSchema, Arrays.asList(record));
+    assert (result.size() == 1);
+    return result.get(0);
+  }
+
+  private List<GenericRecord> writeAndRead(TestSchema testSchema, List<GenericRecord> records) throws IOException {
+    File tmp = File.createTempFile(getClass().getSimpleName(), ".tmp");
+    tmp.deleteOnExit();
+    tmp.delete();
+    Path path = new Path(tmp.getPath());
+
+    try (ParquetWriter<GenericRecord> writer =
+        new TestWriterBuilder(path).withFileType(testSchema.writeSchema).build()) {
+      for (GenericRecord record : records) {
+        writer.write(record);
+      }
+    }
+
+    Configuration conf = new Configuration();
+    // We need to set an explicit read schema because Avro wrote the shredding schema as the Avro
+    // schema in the write, and it will use that by default. If we write using a proper shredding
+    // writer, the Avro schema should just contain a <metadata, value> record, and we won't need this.

Review Comment:
   Is this out of date for this test suite?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130557245


##########
parquet-avro/src/test/java/org/apache/parquet/avro/AvroTestUtil.java:
##########
@@ -129,4 +130,36 @@ public static Configuration conf(String name, boolean value) {
     conf.setBoolean(name, value);
     return conf;
   }
+
+  /**
+   * Assert that to Variant values are logically equivalent.
+   * E.g. object fields may be ordered differently in the binary.

Review Comment:
   Do you mean values? The fields themselves must have the same order because the ID list must be in lexicographical order with respect to the keys.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130577887


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -848,4 +848,73 @@ static HashMap<String, Integer> getMetadataMap(ByteBuffer metadata) {
     }
     return result;
   }
+
+  /**
+   * Computes the actual size (in bytes) of the Variant value at `value[pos...]`
+   * @param value The Variant value
+   * @return The size (in bytes) of the Variant value
+   */
+  public static int valueSize(ByteBuffer value) {
+    int pos = value.position();
+    checkIndex(pos, value.limit());

Review Comment:
   This is guaranteed by `Buffer`, the parent of `ByteBuffer`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130724487


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.

Review Comment:
   Oops, yes, this comment had been on the `variant()` functions below, and ended up in the wrong place.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130817925


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());
+    }
+
+    // Get the element type from the array schema
+    GroupType listType = arrayType.getType(0).asGroupType();
+    Type elementType = listType.getType(0);
+
+    recordConsumer.startGroup();
+    recordConsumer.startField(arrayType.getFieldName(0), 0);
+    // Write each array element
+    for (int i = 0; i < variant.numArrayElements(); i++) {
+      recordConsumer.startGroup();
+      recordConsumer.startField("element", 0);
+
+      recordConsumer.startGroup();

Review Comment:
   `// element group`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130841338


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;

Review Comment:
   I think we need a couple of different decimals in our tests to catch this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155714889


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  private static final String LIST_REPEATED_NAME = "list";
+  private static final String LIST_ELEMENT_NAME = "element";
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField, value)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  // Return true if the logical type is a decimal with the same scale as the provided value, with enough
+  // precision to hold the value. The provided value must be a decimal.
+  private boolean compatibleDecimalType(Variant value, LogicalTypeAnnotation logicalType) {
+    if (!(logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation)) {
+      return false;
+    }
+    LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalType =
+        (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+
+    BigDecimal decimal = value.getDecimal();
+    return decimal.scale() == decimalType.getScale() && decimal.precision() <= decimalType.getPrecision();
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField, Variant value) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                          .isSigned()));

Review Comment:
   Doesn't this also need to check that the width is 64?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155712243


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {
+        Object valueObj = model.getField(record, avroField.name(), index);
+        if (valueObj instanceof byte[]) {
+          value = ByteBuffer.wrap((byte[]) valueObj);
+        } else {
+          value = (ByteBuffer) valueObj;
+        }
+      } else if (fieldType.getName() == "metadata") {
+        Object metadataObj = model.getField(record, avroField.name(), index);
+        if (metadataObj instanceof byte[]) {

Review Comment:
   Looks like this was only done for metadata and not for value. Since it matches bytes, I don't really mind allowing both, but it does make this longer than necessary.
   
   I'd also recommend using `Preconditions.checkArgument` rather than using a separate `if` that throws a generic `RuntimeException`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155872154


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  private static final String LIST_REPEATED_NAME = "list";
+  private static final String LIST_ELEMENT_NAME = "element";
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField, value)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  // Return true if the logical type is a decimal with the same scale as the provided value, with enough
+  // precision to hold the value. The provided value must be a decimal.
+  private boolean compatibleDecimalType(Variant value, LogicalTypeAnnotation logicalType) {
+    if (!(logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation)) {
+      return false;
+    }
+    LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalType =
+        (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+
+    BigDecimal decimal = value.getDecimal();
+    return decimal.scale() == decimalType.getScale() && decimal.precision() <= decimalType.getPrecision();
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField, Variant value) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                          .isSigned()));

Review Comment:
   I don't think the [spec](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md) allows any width other than 64 to be annotated onto an INT64, but I can add it for consistency/readability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2132353714


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());

Review Comment:
   I guess I could, but I don't really understand the benefit. I like that the current version makes it clear that the entire write shares a record consumer and metadata.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130716020


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {

Review Comment:
   Strings can't be compared with `==` because it checks identity. For strings you need to use `"value".equals(fieldType.getName())`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130721189


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {

Review Comment:
   Looks like this is working because the strings used by tests are also interned constants.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130642128


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {
+        Object valueObj = model.getField(record, avroField.name(), index);
+        if (valueObj instanceof byte[]) {
+          value = ByteBuffer.wrap((byte[]) valueObj);
+        } else {
+          value = (ByteBuffer) valueObj;
+        }
+      } else if (fieldType.getName() == "metadata") {
+        Object metadataObj = model.getField(record, avroField.name(), index);
+        if (metadataObj instanceof byte[]) {

Review Comment:
   Why is the type passed in here not known? I think that the representation used for metadata and value should use (and validate) an expected type. Probably `ByteBuffer`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2100529052


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static MessageType readSchema = parquetSchema(variantGroup);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(variantGroup);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(readSchema);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataRawBytes();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataRawBytes();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataRawBytes());
+    vRecord.put(1, v.getValueRawBytes());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(readSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      // assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+      if (!((GenericRecord) record.get(1)).get(1).equals(((GenericRecord) actual.get(1)).get(1))) {
+        assertTrue(false);
+      }
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant v : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();
+      for (int i = 0; i < VARIANTS.length; i++) {
+        expected.add(createRecord(i, VARIANTS[i]));
+      }
+
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      List<GenericRecord> actual = writeAndRead(testSchema, expected);
+      // TODO: CHECK RESULTS

Review Comment:
   Oops, added the checks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130702144


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {
+        Object valueObj = model.getField(record, avroField.name(), index);
+        if (valueObj instanceof byte[]) {
+          value = ByteBuffer.wrap((byte[]) valueObj);
+        } else {
+          value = (ByteBuffer) valueObj;
+        }
+      } else if (fieldType.getName() == "metadata") {
+        Object metadataObj = model.getField(record, avroField.name(), index);
+        if (metadataObj instanceof byte[]) {
+          metadata = ByteBuffer.wrap((byte[]) metadataObj);
+        } else {
+          metadata = (ByteBuffer) metadataObj;
+        }
+      } else {
+        binarySchema = false;
+        break;
+      }
+    }
+
+    if (binarySchema) {
+      VariantValueWriter.write(recordConsumer, schema, new Variant(value, metadata));
+    } else {
+      // If the schema was something other than value and metaadata, treat the value as a non-variant record.

Review Comment:
   Sounds good. My thought was that someone might want to write data that has already been shredded - i.e. annotate the Parquet type as Variant, but directly write using an Avro record that already matches the shredded schema. But if there's a real use case for that, and no other workaround, it can always be added later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130713424


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {
+        Object valueObj = model.getField(record, avroField.name(), index);
+        if (valueObj instanceof byte[]) {
+          value = ByteBuffer.wrap((byte[]) valueObj);
+        } else {
+          value = (ByteBuffer) valueObj;
+        }
+      } else if (fieldType.getName() == "metadata") {
+        Object metadataObj = model.getField(record, avroField.name(), index);
+        if (metadataObj instanceof byte[]) {

Review Comment:
   I was just copying the code used for writing `BYTES`: https://github.com/apache/parquet-java/blob/master/parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java#L357. But I can enforce that it's always `ByteBuffer`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130791522


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());

Review Comment:
   Use `Preconditions.checkArgument`.
   
   Also, error messages are typically more direct, rather than using "but got". A better message is `Cannot write variant type=%s as array`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130825402


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;

Review Comment:
   Doesn't this need to check the decimal scale?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130785684


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;

Review Comment:
   This needs to also check that the logical type is signed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130637011


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {
+        Object valueObj = model.getField(record, avroField.name(), index);
+        if (valueObj instanceof byte[]) {
+          value = ByteBuffer.wrap((byte[]) valueObj);
+        } else {
+          value = (ByteBuffer) valueObj;
+        }
+      } else if (fieldType.getName() == "metadata") {
+        Object metadataObj = model.getField(record, avroField.name(), index);
+        if (metadataObj instanceof byte[]) {
+          metadata = ByteBuffer.wrap((byte[]) metadataObj);
+        } else {
+          metadata = (ByteBuffer) metadataObj;
+        }
+      } else {
+        binarySchema = false;
+        break;
+      }
+    }
+
+    if (binarySchema) {
+      VariantValueWriter.write(recordConsumer, schema, new Variant(value, metadata));
+    } else {
+      // If the schema was something other than value and metaadata, treat the value as a non-variant record.

Review Comment:
   I think that this should be a failure. `writeVariantFields` is called when there is a variant logical type annotation. If the record is not actually a variant then it is better to fail quickly instead of trying to recover by treating it like a record and then failing for some unknown reason later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130563151


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -96,7 +96,7 @@ public ByteBuffer encodedValue() {
    */
   public void appendEncodedValue(ByteBuffer value) {
     onAppend();
-    int size = value.remaining();
+    int size = VariantUtil.valueSize(value);

Review Comment:
   Yeah, I think this is a good change if we are not going to guarantee that the ByteBuffer is limited to the current value. That is an assumption that we make in Iceberg, but not here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2132186030


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueBuffer();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder(new ImmutableMetadata(metadata));
+    appendValue.accept(builder);
+    return builder.build().getValueBuffer();
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static final GroupType UNSHREDDED_GROUP = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static final MessageType READ_SCHEMA = parquetSchema(UNSHREDDED_GROUP);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(UNSHREDDED_GROUP);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(READ_SCHEMA);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataBuffer();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataBuffer();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {

Review Comment:
   When I originally copied the tests, Iceberg didn't have array support, so I just added my own tests. I'll add the cases from the Iceberg tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130686074


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueBuffer();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder(new ImmutableMetadata(metadata));
+    appendValue.accept(builder);
+    return builder.build().getValueBuffer();
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static final GroupType UNSHREDDED_GROUP = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static final MessageType READ_SCHEMA = parquetSchema(UNSHREDDED_GROUP);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(UNSHREDDED_GROUP);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(READ_SCHEMA);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataBuffer();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataBuffer();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataBuffer());
+    vRecord.put(1, v.getValueBuffer());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(READ_SCHEMA, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      Variant actualV = new Variant((ByteBuffer) ((GenericRecord) actual.get(1)).get(1), (ByteBuffer)
+          ((GenericRecord) actual.get(1)).get(0));
+      AvroTestUtil.assertEquivalent(v, actualV);
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant valueForSchema : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();
+      for (int i = 0; i < VARIANTS.length; i++) {
+        expected.add(createRecord(i, VARIANTS[i]));
+      }
+
+      MessageType writeSchema = shreddingSchema(valueForSchema);
+      TestSchema testSchema = new TestSchema(writeSchema, READ_SCHEMA);
+
+      List<GenericRecord> actual = writeAndRead(testSchema, expected);
+      assertEquals(actual.size(), VARIANTS.length);

Review Comment:
   I think it would be more clear if this looped over `expected`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130649582


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.

Review Comment:
   This doesn't really discard metadata. The `Variant` returned has both the metadata and value buffers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130668145


##########
parquet-avro/src/test/java/org/apache/parquet/avro/AvroTestUtil.java:
##########
@@ -129,4 +130,36 @@ public static Configuration conf(String name, boolean value) {
     conf.setBoolean(name, value);
     return conf;
   }
+
+  /**
+   * Assert that to Variant values are logically equivalent.
+   * E.g. object fields may be ordered differently in the binary.

Review Comment:
   Yes, that's what I meant. Is `values in an object may be ordered differently` better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155872154


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  private static final String LIST_REPEATED_NAME = "list";
+  private static final String LIST_ELEMENT_NAME = "element";
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField, value)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  // Return true if the logical type is a decimal with the same scale as the provided value, with enough
+  // precision to hold the value. The provided value must be a decimal.
+  private boolean compatibleDecimalType(Variant value, LogicalTypeAnnotation logicalType) {
+    if (!(logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation)) {
+      return false;
+    }
+    LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalType =
+        (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+
+    BigDecimal decimal = value.getDecimal();
+    return decimal.scale() == decimalType.getScale() && decimal.precision() <= decimalType.getPrecision();
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField, Variant value) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                          .isSigned()));

Review Comment:
   I don't think the spec allows any width other than 64 to be annotated onto an INT64, but I can add it for consistency/readability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155718368


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  private static final String LIST_REPEATED_NAME = "list";
+  private static final String LIST_ELEMENT_NAME = "element";
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField, value)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  // Return true if the logical type is a decimal with the same scale as the provided value, with enough
+  // precision to hold the value. The provided value must be a decimal.
+  private boolean compatibleDecimalType(Variant value, LogicalTypeAnnotation logicalType) {
+    if (!(logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation)) {
+      return false;
+    }
+    LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalType =
+        (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+
+    BigDecimal decimal = value.getDecimal();
+    return decimal.scale() == decimalType.getScale() && decimal.precision() <= decimalType.getPrecision();
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField, Variant value) {

Review Comment:
   Can be static.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099014659


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static MessageType readSchema = parquetSchema(variantGroup);

Review Comment:
   This can be final and since it's static it should use `ALL_CAPS`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099021924


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  // We store a reference to the metdata so that we can lazily build metadataMap if needed.
+  // It could be relatively expensive to build, and if everything shreds perfectly, we won't need it.

Review Comment:
   Is this true? I think that metadata should be the same regardless of shredding so that it does not need to be rebuilt and re-encoded. That way we also don't need to change IDs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099020131


##########
parquet-variant/src/main/java/org/apache/parquet/variant/Variant.java:
##########
@@ -61,6 +61,14 @@ public Variant(ByteBuffer value, ByteBuffer metadata) {
     }
   }
 
+  public ByteBuffer getValueRawBytes() {

Review Comment:
   "raw" isn't very clear. What about `encodedValue` and `encodedMetadata`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2101043579


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -96,7 +96,7 @@ public ByteBuffer encodedValue() {
    */
   public void appendEncodedValue(ByteBuffer value) {
     onAppend();
-    int size = value.remaining();
+    int size = VariantUtil.valueSize(value);

Review Comment:
   Without this change, we would write the entire `ByteBuffer`, which could go to the end of the buffer. The written value would technically be correct (since a reader would ignore the remaining bytes), but obviously very inefficient. An alternative to limiting the size here is to do it in the `getValueBuffer()` method. But depending on what the caller plans to do with the buffer, the extra bytes might not matter, and we'd be parsing the value unnecessarily.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130625524


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {
+        Object valueObj = model.getField(record, avroField.name(), index);
+        if (valueObj instanceof byte[]) {
+          value = ByteBuffer.wrap((byte[]) valueObj);
+        } else {
+          value = (ByteBuffer) valueObj;
+        }
+      } else if (fieldType.getName() == "metadata") {
+        Object metadataObj = model.getField(record, avroField.name(), index);
+        if (metadataObj instanceof byte[]) {
+          metadata = ByteBuffer.wrap((byte[]) metadataObj);
+        } else {
+          metadata = (ByteBuffer) metadataObj;
+        }
+      } else {
+        binarySchema = false;
+        break;
+      }
+    }
+
+    if (binarySchema) {
+      VariantValueWriter.write(recordConsumer, schema, new Variant(value, metadata));
+    } else {
+      // If the schema was something other than value and metaadata, treat the value as a non-variant record.

Review Comment:
   Typo: `metaa`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130800249


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());
+    }
+
+    // Get the element type from the array schema
+    GroupType listType = arrayType.getType(0).asGroupType();

Review Comment:
   Do you also want to assert that the structure is a 3-level list rather than just accessing fields?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130791522


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());

Review Comment:
   Use `Preconditions.checkArgument`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130770655


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {

Review Comment:
   I've verified that the tests are actually testing what is intended. I'd prefer to have the same array cases as Iceberg, but otherwise this looks mostly good.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099163036


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static MessageType readSchema = parquetSchema(variantGroup);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(variantGroup);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(readSchema);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataRawBytes();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataRawBytes();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataRawBytes());
+    vRecord.put(1, v.getValueRawBytes());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(readSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      // assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+      if (!((GenericRecord) record.get(1)).get(1).equals(((GenericRecord) actual.get(1)).get(1))) {

Review Comment:
   Oops, the commented out line is the right assert. I think I was doing something in the debugger and forgot to restore it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2100528517


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)

Review Comment:
   Changed to UNSHREDDED_GROUP.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2098948759


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  // We store a reference to the metdata so that we can lazily build metadataMap if needed.
+  // It could be relatively expensive to build, and if everything shreds perfectly, we won't need it.
+  private ByteBuffer metadata;
+  private HashMap<String, Integer> metadataMap = null;

Review Comment:
   TODO: replace with an ImmutableMetadata. I think we still want to create is lazily.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  // We store a reference to the metdata so that we can lazily build metadataMap if needed.
+  // It could be relatively expensive to build, and if everything shreds perfectly, we won't need it.
+  private ByteBuffer metadata;
+  private HashMap<String, Integer> metadataMap = null;

Review Comment:
   TODO: replace with an ImmutableMetadata. I think we still want to create it lazily.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2161678053


##########
parquet-avro/src/main/java/org/apache/parquet/avro/AvroWriteSupport.java:
##########
@@ -181,9 +184,58 @@ public void write(T record) {
   }
 
   private void writeRecord(GroupType schema, Schema avroSchema, Object record) {
-    recordConsumer.startGroup();
-    writeRecordFields(schema, avroSchema, record);
-    recordConsumer.endGroup();
+    if (schema.getLogicalTypeAnnotation() instanceof LogicalTypeAnnotation.VariantLogicalTypeAnnotation) {
+      writeVariantFields(schema, avroSchema, record);
+    } else {
+      recordConsumer.startGroup();
+      writeRecordFields(schema, avroSchema, record);
+      recordConsumer.endGroup();
+    }
+  }
+
+  private void writeVariantFields(GroupType schema, Schema avroSchema, Object record) {
+    List<Type> fields = schema.getFields();
+    List<Schema.Field> avroFields = avroSchema.getFields();
+    boolean binarySchema = true;
+    ByteBuffer metadata = null;
+    ByteBuffer value = null;
+    // Extract the value and metadata binary.
+    for (int index = 0; index < avroFields.size(); index++) {
+      Schema.Field avroField = avroFields.get(index);
+      Schema fieldSchema = AvroSchemaConverter.getNonNull(avroField.schema());
+      if (!fieldSchema.getType().equals(Schema.Type.BYTES)) {
+        binarySchema = false;
+        break;
+      }
+      Type fieldType = fields.get(index);
+      if (fieldType.getName() == "value") {
+        Object valueObj = model.getField(record, avroField.name(), index);
+        if (valueObj instanceof byte[]) {
+          value = ByteBuffer.wrap((byte[]) valueObj);
+        } else {
+          value = (ByteBuffer) valueObj;
+        }
+      } else if (fieldType.getName() == "metadata") {
+        Object metadataObj = model.getField(record, avroField.name(), index);
+        if (metadataObj instanceof byte[]) {
+          metadata = ByteBuffer.wrap((byte[]) metadataObj);
+        } else {
+          metadata = (ByteBuffer) metadataObj;
+        }
+      } else {
+        binarySchema = false;
+        break;
+      }
+    }
+
+    if (binarySchema) {
+      VariantValueWriter.write(recordConsumer, schema, new Variant(value, metadata));
+    } else {
+      // If the schema was something other than value and metaadata, treat the value as a non-variant record.

Review Comment:
   Hi @rdblue, I realized that this was needed in order to allow the tests in TestReadVariant.java to work, because those tests are doing what I described in this comment: writing already-shredded data with the logical annotation on the top-level struct.
   
   I pushed a commit to explicitly check for the cases of having exactly three fields in the schema, or having two fields with `typed_value`, which is also valid, but otherwise leave in the stricter checking that was added based on this comment. Let me know if you think this is okay, or if you prefer a different approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#issuecomment-2998005007

   Running tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155706277


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  // We store a reference to the metdata so that we can lazily build metadataMap if needed.
+  // It could be relatively expensive to build, and if everything shreds perfectly, we won't need it.

Review Comment:
   If I understand correctly, you're saying that this is avoiding _deserializing_ the object's metadata. The metadata may need to be read into a map in order to re-encode the residual object. I think that makes sense.
   
   The residual object is a strict subset of the original object's fields (just the remaining unshredded ones) so the residual's fields will be in the same order as the original and will use the same IDs because the metadata doesn't change. But the API for actually building the residual object uses the field names rather than static IDs. So you can either build the residual with the normal API or you could have a special case in the builder where the caller is responsible for supplying the IDs instead of names. I wouldn't bother with that optimization in this version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155718562


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  private static final String LIST_REPEATED_NAME = "list";
+  private static final String LIST_ELEMENT_NAME = "element";
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField, value)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  // Return true if the logical type is a decimal with the same scale as the provided value, with enough
+  // precision to hold the value. The provided value must be a decimal.
+  private boolean compatibleDecimalType(Variant value, LogicalTypeAnnotation logicalType) {
+    if (!(logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation)) {
+      return false;
+    }
+    LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalType =
+        (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+
+    BigDecimal decimal = value.getDecimal();
+    return decimal.scale() == decimalType.getScale() && decimal.precision() <= decimalType.getPrecision();
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField, Variant value) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                          .isSigned()));
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && compatibleDecimalType(value, logicalType);
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && compatibleDecimalType(value, logicalType);
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && compatibleDecimalType(value, logicalType);
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {

Review Comment:
   Type is not used.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130677616


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueBuffer();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder(new ImmutableMetadata(metadata));
+    appendValue.accept(builder);
+    return builder.build().getValueBuffer();
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static final GroupType UNSHREDDED_GROUP = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static final MessageType READ_SCHEMA = parquetSchema(UNSHREDDED_GROUP);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(UNSHREDDED_GROUP);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(READ_SCHEMA);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataBuffer();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataBuffer();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return

Review Comment:
   Omit `@return` if there isn't relevant context to add.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130673927


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueBuffer();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder(new ImmutableMetadata(metadata));
+    appendValue.accept(builder);
+    return builder.build().getValueBuffer();
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static final GroupType UNSHREDDED_GROUP = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static final MessageType READ_SCHEMA = parquetSchema(UNSHREDDED_GROUP);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(UNSHREDDED_GROUP);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(READ_SCHEMA);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataBuffer();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataBuffer();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {

Review Comment:
   Looks like this deviates from the Iceberg array tests, which use an array of strings, an array of strings and an int, an array of arrays of strings, and then some nested objects. Why does this skip the simpler array cases?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130615441


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -848,4 +848,73 @@ static HashMap<String, Integer> getMetadataMap(ByteBuffer metadata) {
     }
     return result;
   }
+
+  /**
+   * Computes the actual size (in bytes) of the Variant value at `value[pos...]`
+   * @param value The Variant value
+   * @return The size (in bytes) of the Variant value
+   */
+  public static int valueSize(ByteBuffer value) {
+    int pos = value.position();
+    checkIndex(pos, value.limit());
+    int basicType = value.get(pos) & BASIC_TYPE_MASK;
+    int typeInfo = (value.get(pos) >> BASIC_TYPE_BITS) & PRIMITIVE_TYPE_MASK;
+    switch (basicType) {

Review Comment:
   The logic in this function look correct to me. It may be helpful to note in a comment that the size includes the header byte.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130804463


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());
+    }
+
+    // Get the element type from the array schema
+    GroupType listType = arrayType.getType(0).asGroupType();
+    Type elementType = listType.getType(0);
+
+    recordConsumer.startGroup();

Review Comment:
   Comments would help here. The Avro list writer has `// group wrapper (original type LIST)`, which would probably be `// list field, annotated as LIST` now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130808457


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());
+    }
+
+    // Get the element type from the array schema
+    GroupType listType = arrayType.getType(0).asGroupType();
+    Type elementType = listType.getType(0);
+
+    recordConsumer.startGroup();
+    recordConsumer.startField(arrayType.getFieldName(0), 0);

Review Comment:
   It would be a bit better to store the repeated type and access its name here: `repeatedType.getName()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099160474


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  // We store a reference to the metdata so that we can lazily build metadataMap if needed.
+  // It could be relatively expensive to build, and if everything shreds perfectly, we won't need it.

Review Comment:
   I'm not referring here to building metadata, but using the existing metadata to build a map from keys to metadata IDs. We need it in the case where we need to build a residual object in `value` if an object contains keys that aren't in `typed_value`. In any case, with the latest version of the reader PR, this happens in the `ImmutableMetadata` constructor, so the code will just defer creating `ImmutableMetadata` until it's actually needed.
   
   The truth is, I don't think we strictly need any metadata at all for the writer: it's only necessary because we're using the `VariantObjectBuilder` interface to construct the leftover objects in `value`, and the object builder requires the key String to be provided by name, not ID, so that it can sort objects by key when it finalizes an object. If we assume the input is valid (i.e. already sorted), we could create a short-circuiting version of `VariantObjectBuilder` that operates entirely in terms of the IDs in the variant, in which case we'd never need to reference the metadata. Let me know if you think it's worth extending the object builder with that change. I think it's just a performance benefit, and maybe we don't care that much about that if this is mainly a reference implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099017806


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static MessageType readSchema = parquetSchema(variantGroup);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(variantGroup);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(readSchema);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataRawBytes();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataRawBytes();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataRawBytes());
+    vRecord.put(1, v.getValueRawBytes());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(readSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      // assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+      if (!((GenericRecord) record.get(1)).get(1).equals(((GenericRecord) actual.get(1)).get(1))) {

Review Comment:
   This seems odd. What's going on here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2101043579


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -96,7 +96,7 @@ public ByteBuffer encodedValue() {
    */
   public void appendEncodedValue(ByteBuffer value) {
     onAppend();
-    int size = value.remaining();
+    int size = VariantUtil.valueSize(value);

Review Comment:
   Without this change, we would write to the limit() of `ByteBuffer`, which could go well past the value being written. The written value would technically be correct (since a reader would ignore the remaining bytes), but obviously very inefficient. An alternative to limiting the size here is to do it in the `getValueBuffer()` method. But depending on what the caller plans to do with the buffer, the extra bytes might not matter, and we'd be parsing the value unnecessarily.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155908438


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());

Review Comment:
   Good point, I'll remove `recordConsumer` from the argument list of the non-static methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#issuecomment-2993096758

   Thanks for the updates, @cashmand! I think this looks good to go. Running the tests again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155720373


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;

Review Comment:
   This can be final.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130816455


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());
+    }
+
+    // Get the element type from the array schema
+    GroupType listType = arrayType.getType(0).asGroupType();
+    Type elementType = listType.getType(0);
+
+    recordConsumer.startGroup();
+    recordConsumer.startField(arrayType.getFieldName(0), 0);
+    // Write each array element
+    for (int i = 0; i < variant.numArrayElements(); i++) {
+      recordConsumer.startGroup();
+      recordConsumer.startField("element", 0);

Review Comment:
   Could you use a constant like in `AvroWriteSupport`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130813892


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    if (variant.getType() != Variant.Type.ARRAY) {
+      throw new IllegalArgumentException("Expected array type but got: " + variant.getType());
+    }
+
+    // Get the element type from the array schema
+    GroupType listType = arrayType.getType(0).asGroupType();
+    Type elementType = listType.getType(0);
+
+    recordConsumer.startGroup();
+    recordConsumer.startField(arrayType.getFieldName(0), 0);
+    // Write each array element
+    for (int i = 0; i < variant.numArrayElements(); i++) {
+      recordConsumer.startGroup();

Review Comment:
   `// repeated group`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2132356669


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation);
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;

Review Comment:
   Yikes, good catch! I added more decimal test cases, which catches this issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130855957


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());

Review Comment:
   This is not a strong opinion, but it looks like the only reason why there is a writer instance is to carry the metadata buffer. Since all of these are internal methods that aren't exposed, I'd rather just use method args to pass the record consumer and metadata.
   
   I guess this also lazily instantiates an `ImmutableMetadata`, but that doesn't seem like a high cost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130694726


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueBuffer();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder(new ImmutableMetadata(metadata));
+    appendValue.accept(builder);
+    return builder.build().getValueBuffer();
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static final GroupType UNSHREDDED_GROUP = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static final MessageType READ_SCHEMA = parquetSchema(UNSHREDDED_GROUP);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(UNSHREDDED_GROUP);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(READ_SCHEMA);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataBuffer();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataBuffer();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataBuffer());
+    vRecord.put(1, v.getValueBuffer());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(READ_SCHEMA, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      Variant actualV = new Variant((ByteBuffer) ((GenericRecord) actual.get(1)).get(1), (ByteBuffer)
+          ((GenericRecord) actual.get(1)).get(0));
+      AvroTestUtil.assertEquivalent(v, actualV);
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant valueForSchema : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();
+      for (int i = 0; i < VARIANTS.length; i++) {
+        expected.add(createRecord(i, VARIANTS[i]));
+      }
+
+      MessageType writeSchema = shreddingSchema(valueForSchema);
+      TestSchema testSchema = new TestSchema(writeSchema, READ_SCHEMA);
+
+      List<GenericRecord> actual = writeAndRead(testSchema, expected);
+      assertEquals(actual.size(), VARIANTS.length);
+      for (int i = 0; i < VARIANTS.length; i++) {
+        Variant actualV =
+            new Variant((ByteBuffer) ((GenericRecord) actual.get(i).get(1)).get(1), (ByteBuffer)
+                ((GenericRecord) actual.get(i).get(1)).get(0));
+        AvroTestUtil.assertEquivalent(VARIANTS[i], actualV);
+      }
+    }
+  }
+
+  // Write schema contains the full shredding schema. Read schema should just be a value/metadata pair.
+  private static class TestSchema {
+    MessageType writeSchema;
+    MessageType readSchema;
+
+    TestSchema(MessageType writeSchema, MessageType readSchema) {
+      this.writeSchema = writeSchema;
+      this.readSchema = readSchema;
+    }
+  }
+
+  /**
+   * This is a custom Parquet writer builder that injects a specific Parquet schema and then uses
+   * the Avro object model. This ensures that the Parquet file's schema is exactly what was passed.
+   */
+  private static class TestWriterBuilder extends ParquetWriter.Builder<GenericRecord, TestWriterBuilder> {
+    private MessageType schema = null;
+
+    protected TestWriterBuilder(Path path) {
+      super(path);
+    }
+
+    TestWriterBuilder withFileType(MessageType schema) {
+      this.schema = schema;
+      return self();
+    }
+
+    @Override
+    protected TestWriterBuilder self() {
+      return this;
+    }
+
+    @Override
+    protected WriteSupport<GenericRecord> getWriteSupport(Configuration conf) {
+      return new AvroWriteSupport<>(schema, new AvroSchemaConverter().convert(schema), GenericData.get());
+    }
+  }
+
+  GenericRecord writeAndRead(TestSchema testSchema, GenericRecord record) throws IOException {
+    List<GenericRecord> result = writeAndRead(testSchema, Arrays.asList(record));
+    assert (result.size() == 1);

Review Comment:
   This isn't the same as a test assertion. In tests, prefer using `assertEquals(1, result.size())`.
   
   In fact, `assert` is not used in most Java projects in our space.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130694726


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueBuffer();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder(new ImmutableMetadata(metadata));
+    appendValue.accept(builder);
+    return builder.build().getValueBuffer();
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static final GroupType UNSHREDDED_GROUP = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static final MessageType READ_SCHEMA = parquetSchema(UNSHREDDED_GROUP);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(UNSHREDDED_GROUP);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(READ_SCHEMA);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataBuffer();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataBuffer();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataBuffer());
+    vRecord.put(1, v.getValueBuffer());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(READ_SCHEMA, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      Variant actualV = new Variant((ByteBuffer) ((GenericRecord) actual.get(1)).get(1), (ByteBuffer)
+          ((GenericRecord) actual.get(1)).get(0));
+      AvroTestUtil.assertEquivalent(v, actualV);
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant valueForSchema : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();
+      for (int i = 0; i < VARIANTS.length; i++) {
+        expected.add(createRecord(i, VARIANTS[i]));
+      }
+
+      MessageType writeSchema = shreddingSchema(valueForSchema);
+      TestSchema testSchema = new TestSchema(writeSchema, READ_SCHEMA);
+
+      List<GenericRecord> actual = writeAndRead(testSchema, expected);
+      assertEquals(actual.size(), VARIANTS.length);
+      for (int i = 0; i < VARIANTS.length; i++) {
+        Variant actualV =
+            new Variant((ByteBuffer) ((GenericRecord) actual.get(i).get(1)).get(1), (ByteBuffer)
+                ((GenericRecord) actual.get(i).get(1)).get(0));
+        AvroTestUtil.assertEquivalent(VARIANTS[i], actualV);
+      }
+    }
+  }
+
+  // Write schema contains the full shredding schema. Read schema should just be a value/metadata pair.
+  private static class TestSchema {
+    MessageType writeSchema;
+    MessageType readSchema;
+
+    TestSchema(MessageType writeSchema, MessageType readSchema) {
+      this.writeSchema = writeSchema;
+      this.readSchema = readSchema;
+    }
+  }
+
+  /**
+   * This is a custom Parquet writer builder that injects a specific Parquet schema and then uses
+   * the Avro object model. This ensures that the Parquet file's schema is exactly what was passed.
+   */
+  private static class TestWriterBuilder extends ParquetWriter.Builder<GenericRecord, TestWriterBuilder> {
+    private MessageType schema = null;
+
+    protected TestWriterBuilder(Path path) {
+      super(path);
+    }
+
+    TestWriterBuilder withFileType(MessageType schema) {
+      this.schema = schema;
+      return self();
+    }
+
+    @Override
+    protected TestWriterBuilder self() {
+      return this;
+    }
+
+    @Override
+    protected WriteSupport<GenericRecord> getWriteSupport(Configuration conf) {
+      return new AvroWriteSupport<>(schema, new AvroSchemaConverter().convert(schema), GenericData.get());
+    }
+  }
+
+  GenericRecord writeAndRead(TestSchema testSchema, GenericRecord record) throws IOException {
+    List<GenericRecord> result = writeAndRead(testSchema, Arrays.asList(record));
+    assert (result.size() == 1);

Review Comment:
   This isn't the same as a test assertion. In tests, prefer using `assertEquals(1, result.size())`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2130687824


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,518 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.ImmutableMetadata;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueBuffer();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder(new ImmutableMetadata(metadata));
+    appendValue.accept(builder);
+    return builder.build().getValueBuffer();
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static final GroupType UNSHREDDED_GROUP = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static final MessageType READ_SCHEMA = parquetSchema(UNSHREDDED_GROUP);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(UNSHREDDED_GROUP);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(READ_SCHEMA);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataBuffer();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataBuffer();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataBuffer());
+    vRecord.put(1, v.getValueBuffer());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(READ_SCHEMA, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, READ_SCHEMA);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      Variant actualV = new Variant((ByteBuffer) ((GenericRecord) actual.get(1)).get(1), (ByteBuffer)
+          ((GenericRecord) actual.get(1)).get(0));
+      AvroTestUtil.assertEquivalent(v, actualV);
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant valueForSchema : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();

Review Comment:
   This can be moved outside the `valueForSchema` loop because it doesn't change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2099017979


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static MessageType readSchema = parquetSchema(variantGroup);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(variantGroup);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(readSchema);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataRawBytes();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataRawBytes();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataRawBytes());
+    vRecord.put(1, v.getValueRawBytes());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(readSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      // assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+      if (!((GenericRecord) record.get(1)).get(1).equals(((GenericRecord) actual.get(1)).get(1))) {
+        assertTrue(false);
+      }
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant v : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();
+      for (int i = 0; i < VARIANTS.length; i++) {
+        expected.add(createRecord(i, VARIANTS[i]));
+      }
+
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      List<GenericRecord> actual = writeAndRead(testSchema, expected);
+      // TODO: CHECK RESULTS

Review Comment:
   Looks like this is needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2098948759


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,310 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.util.HashMap;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  // We store a reference to the metdata so that we can lazily build metadataMap if needed.
+  // It could be relatively expensive to build, and if everything shreds perfectly, we won't need it.
+  private ByteBuffer metadata;
+  private HashMap<String, Integer> metadataMap = null;

Review Comment:
   TODO: replace with an ImmutableMetadata. I think we still want to create it lazily for the reason described above.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] Implement Variant parquet writer [parquet-java]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2100527909


##########
parquet-avro/src/test/java/org/apache/parquet/avro/TestWriteVariant.java:
##########
@@ -0,0 +1,517 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.avro;
+
+import static org.apache.parquet.avro.AvroTestUtil.*;
+import static org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName.*;
+import static org.junit.Assert.*;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import java.util.*;
+import java.util.function.Consumer;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.DirectWriterTest;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.api.WriteSupport;
+import org.apache.parquet.schema.*;
+import org.apache.parquet.schema.LogicalTypeAnnotation.TimeUnit;
+import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName;
+import org.apache.parquet.variant.Variant;
+import org.apache.parquet.variant.VariantArrayBuilder;
+import org.apache.parquet.variant.VariantBuilder;
+import org.apache.parquet.variant.VariantObjectBuilder;
+import org.junit.Test;
+
+public class TestWriteVariant extends DirectWriterTest {
+
+  // Construct a variant, and return the value binary, dropping metadata.
+  private static Variant fullVariant(Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    appendValue.accept(builder);
+    return builder.build();
+  }
+
+  // Return only the byte[], which is usually all we want.
+  private static ByteBuffer variant(Consumer<VariantBuilder> appendValue) {
+    return fullVariant(appendValue).getValueRawBytes();
+  }
+
+  // Returns a value based on building with fixed metadata.
+  private static ByteBuffer variant(ByteBuffer metadata, Consumer<VariantBuilder> appendValue) {
+    VariantBuilder builder = new VariantBuilder();
+    builder.setFixedMetadata(metadata);
+    appendValue.accept(builder);
+    return ByteBuffer.wrap(builder.valueWithoutMetadata());
+  }
+
+  private static ByteBuffer variant(int val) {
+    return variant(b -> b.appendInt(val));
+  }
+
+  private static ByteBuffer variant(long val) {
+    return variant(b -> b.appendLong(val));
+  }
+
+  private static ByteBuffer variant(String s) {
+    return variant(b -> b.appendString(s));
+  }
+
+  private static GroupType variantGroup = Types.buildGroup(Type.Repetition.REQUIRED)
+      .as(LogicalTypeAnnotation.variantType((byte) 1))
+      .required(PrimitiveTypeName.BINARY)
+      .named("metadata")
+      .required(PrimitiveTypeName.BINARY)
+      .named("value")
+      .named("var");
+
+  private static MessageType parquetSchema(GroupType variantGroup) {
+    return Types.buildMessage()
+        .required(INT32)
+        .named("id")
+        .addField(variantGroup)
+        .named("table");
+  }
+
+  private static MessageType readSchema = parquetSchema(variantGroup);
+
+  private static final Schema VARIANT_SCHEMA = new AvroSchemaConverter().convert(variantGroup);
+  private static final Schema SCHEMA = new AvroSchemaConverter().convert(readSchema);
+
+  private ByteBuffer TEST_METADATA;
+  private ByteBuffer TEST_OBJECT;
+  private ByteBuffer SIMILAR_OBJECT;
+  private ByteBuffer TEST_ARRAY;
+  private ByteBuffer SIMILAR_ARRAY;
+  private ByteBuffer EMPTY_OBJECT;
+  private ByteBuffer EMPTY_METADATA = fullVariant(b -> b.appendNull()).getMetadataRawBytes();
+  private Variant[] VARIANTS;
+
+  public TestWriteVariant() throws Exception {
+    TEST_METADATA = fullVariant(b -> {
+          VariantObjectBuilder ob = b.startObject();
+          ob.appendKey("a");
+          ob.appendNull();
+          ob.appendKey("b");
+          ob.appendNull();
+          ob.appendKey("c");
+          ob.appendNull();
+          ob.appendKey("d");
+          ob.appendNull();
+          ob.appendKey("e");
+          ob.appendNull();
+          b.endObject();
+        })
+        .getMetadataRawBytes();
+
+    TEST_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      b.endObject();
+    });
+
+    SIMILAR_OBJECT = variant(TEST_METADATA, b -> {
+      VariantObjectBuilder ob = b.startObject();
+      ob.appendKey("a");
+      ob.appendInt(123456789);
+      ob.appendKey("c");
+      ob.appendString("string");
+      b.endObject();
+    });
+
+    // The first array element defines the schema.
+    TEST_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("a");
+      ob.appendNull();
+      ob.appendKey("d");
+      ob.appendString("iceberg");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    // Change one field name and one type in the first element to change the schema.
+    SIMILAR_ARRAY = variant(TEST_METADATA, b -> {
+      VariantArrayBuilder ab = b.startArray();
+      VariantObjectBuilder ob = ab.startObject();
+      ob.appendKey("c");
+      ob.appendString("iceberg");
+      ob.appendKey("a");
+      ob.appendString("parquet");
+      ab.endObject();
+      ab.appendInt(123);
+      VariantObjectBuilder ob2 = ab.startObject();
+      ob2.appendKey("c");
+      ob2.appendString("hello");
+      ob2.appendKey("d");
+      ob2.appendDate(12345);
+      ab.endObject();
+      b.endArray();
+    });
+
+    EMPTY_OBJECT = variant(TEST_METADATA, b -> {
+      b.startObject();
+      b.endObject();
+    });
+
+    VARIANTS = new Variant[] {
+      fullVariant(b -> b.appendNull()),
+      fullVariant(b -> b.appendBoolean(true)),
+      fullVariant(b -> b.appendBoolean(false)),
+      fullVariant(b -> b.appendByte((byte) 34)),
+      fullVariant(b -> b.appendByte((byte) -34)),
+      fullVariant(b -> b.appendShort((byte) 1234)),
+      fullVariant(b -> b.appendShort((byte) -1234)),
+      fullVariant(b -> b.appendInt(12345)),
+      fullVariant(b -> b.appendInt(-12345)),
+      fullVariant(b -> b.appendLong(9876543210L)),
+      fullVariant(b -> b.appendLong(-9876543210L)),
+      fullVariant(b -> b.appendFloat(10.11F)),
+      fullVariant(b -> b.appendFloat(-10.11F)),
+      fullVariant(b -> b.appendDouble(14.3D)),
+      fullVariant(b -> b.appendDouble(-14.3D)),
+      new Variant(EMPTY_OBJECT, EMPTY_METADATA),
+      new Variant(TEST_OBJECT, TEST_METADATA),
+      new Variant(SIMILAR_OBJECT, TEST_METADATA),
+      new Variant(TEST_ARRAY, TEST_METADATA),
+      new Variant(SIMILAR_ARRAY, TEST_METADATA),
+      fullVariant(b -> b.appendDate(12345)),
+      fullVariant(b -> b.appendDate(-12345)),
+      fullVariant(b -> b.appendTimestampTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNtz(-1234567890L)),
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456.789"))), // decimal4
+      fullVariant(b -> b.appendDecimal(new BigDecimal("123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-123456789.987654321"))), // decimal8
+      fullVariant(b -> b.appendDecimal(new BigDecimal("9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendDecimal(new BigDecimal("-9876543210.123456789"))), // decimal16
+      fullVariant(b -> b.appendBinary(ByteBuffer.wrap(new byte[] {0x0a, 0x0b, 0x0c, 0x0d}))),
+      fullVariant(b -> b.appendString("iceberg")),
+      fullVariant(b -> b.appendTime(1234567890)),
+      fullVariant(b -> b.appendTimestampNanosTz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosTz(-1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(1234567890L)),
+      fullVariant(b -> b.appendTimestampNanosNtz(-1234567890L)),
+      fullVariant(b -> b.appendUUID(UUID.fromString("f24f9b64-81fa-49d1-b74e-8c09a6e31c56")))
+    };
+  }
+
+  /**
+   * Create a record containing a Variant value using the standard schema.
+   * @return
+   */
+  GenericRecord createRecord(int i, Variant v) {
+    GenericRecord vRecord = new GenericData.Record(VARIANT_SCHEMA);
+    vRecord.put(0, v.getMetadataRawBytes());
+    vRecord.put(1, v.getValueRawBytes());
+    GenericRecord record = new GenericData.Record(SCHEMA);
+    record.put(0, i);
+    record.put(1, vRecord);
+    return record;
+  }
+
+  // Tests in this file are based on Iceberg's TestVariantWriters suite.
+  @Test
+  public void testUnshreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      TestSchema testSchema = new TestSchema(readSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+    }
+  }
+
+  @Test
+  public void testShreddedValues() throws IOException {
+    for (Variant v : VARIANTS) {
+      GenericRecord record = createRecord(1, v);
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      GenericRecord actual = writeAndRead(testSchema, record);
+      assertEquals(record.get(0), actual.get(0));
+      assertEquals(((GenericRecord) record.get(1)).get(0), ((GenericRecord) actual.get(1)).get(0));
+      // assertEquals(((GenericRecord) record.get(1)).get(1), ((GenericRecord) actual.get(1)).get(1));
+      if (!((GenericRecord) record.get(1)).get(1).equals(((GenericRecord) actual.get(1)).get(1))) {
+        assertTrue(false);
+      }
+    }
+  }
+
+  @Test
+  public void testMixedShredding() throws IOException {
+    for (Variant v : VARIANTS) {
+      List<GenericRecord> expected = new ArrayList<>();
+      for (int i = 0; i < VARIANTS.length; i++) {
+        expected.add(createRecord(i, VARIANTS[i]));
+      }
+
+      MessageType writeSchema = shreddingSchema(v);
+      TestSchema testSchema = new TestSchema(writeSchema, readSchema);
+
+      List<GenericRecord> actual = writeAndRead(testSchema, expected);
+      // TODO: CHECK RESULTS
+    }
+  }
+
+  // Write schema contains the full shredding schema. Read schema should just be a value/metadata pair.
+  private static class TestSchema {
+    MessageType writeSchema;
+    MessageType readSchema;
+
+    TestSchema(MessageType writeSchema, MessageType readSchema) {
+      this.writeSchema = writeSchema;
+      this.readSchema = readSchema;
+    }
+  }
+
+  /**
+   * This is a custom Parquet writer builder that injects a specific Parquet schema and then uses
+   * the Avro object model. This ensures that the Parquet file's schema is exactly what was passed.
+   */
+  private static class TestWriterBuilder extends ParquetWriter.Builder<GenericRecord, TestWriterBuilder> {
+    private MessageType schema = null;
+
+    protected TestWriterBuilder(Path path) {
+      super(path);
+    }
+
+    TestWriterBuilder withFileType(MessageType schema) {
+      this.schema = schema;
+      return self();
+    }
+
+    @Override
+    protected TestWriterBuilder self() {
+      return this;
+    }
+
+    @Override
+    protected WriteSupport<GenericRecord> getWriteSupport(Configuration conf) {
+      return new AvroWriteSupport<>(schema, new AvroSchemaConverter().convert(schema), GenericData.get());
+    }
+  }
+
+  GenericRecord writeAndRead(TestSchema testSchema, GenericRecord record) throws IOException {
+    List<GenericRecord> result = writeAndRead(testSchema, Arrays.asList(record));
+    assert (result.size() == 1);
+    return result.get(0);
+  }
+
+  private List<GenericRecord> writeAndRead(TestSchema testSchema, List<GenericRecord> records) throws IOException {
+    File tmp = File.createTempFile(getClass().getSimpleName(), ".tmp");
+    tmp.deleteOnExit();
+    tmp.delete();
+    Path path = new Path(tmp.getPath());
+
+    try (ParquetWriter<GenericRecord> writer =
+        new TestWriterBuilder(path).withFileType(testSchema.writeSchema).build()) {
+      for (GenericRecord record : records) {
+        writer.write(record);
+      }
+    }
+
+    Configuration conf = new Configuration();
+    // We need to set an explicit read schema because Avro wrote the shredding schema as the Avro
+    // schema in the write, and it will use that by default. If we write using a proper shredding
+    // writer, the Avro schema should just contain a <metadata, value> record, and we won't need this.

Review Comment:
   Yes, deleted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155716247


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,372 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.nio.ByteBuffer;
+import org.apache.parquet.Preconditions;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  private static final String LIST_REPEATED_NAME = "list";
+  private static final String LIST_ELEMENT_NAME = "element";
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());
+    writer.write(schema, value);
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write a Variant value to a shredded schema. The caller is responsible for calling startGroup()
+   * and endGroup(), and writing metadata if this is the top level of the Variant group.
+   */
+  void write(GroupType schema, Variant value) {
+    Type typedValueField = null;
+    if (schema.containsField("typed_value")) {
+      typedValueField = schema.getType("typed_value");
+    }
+
+    Variant.Type variantType = value.getType();
+
+    // Handle typed_value if present
+    if (isTypeCompatible(variantType, typedValueField, value)) {
+      int typedValueIdx = schema.getFieldIndex("typed_value");
+      recordConsumer.startField("typed_value", typedValueIdx);
+      ByteBuffer residual = null;
+      if (typedValueField.isPrimitive()) {
+        writeScalarValue(recordConsumer, value, typedValueField.asPrimitiveType());
+      } else if (typedValueField.getLogicalTypeAnnotation()
+          instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+        writeArrayValue(recordConsumer, value, typedValueField.asGroupType());
+      } else {
+        residual = writeObjectValue(recordConsumer, value, typedValueField.asGroupType());
+      }
+      recordConsumer.endField("typed_value", typedValueIdx);
+
+      if (residual != null) {
+        int valueIdx = schema.getFieldIndex("value");
+        recordConsumer.startField("value", valueIdx);
+        recordConsumer.addBinary(Binary.fromConstantByteBuffer(residual));
+        recordConsumer.endField("value", valueIdx);
+      }
+    } else {
+      int valueIdx = schema.getFieldIndex("value");
+      recordConsumer.startField("value", valueIdx);
+      recordConsumer.addBinary(Binary.fromReusedByteBuffer(value.getValueBuffer()));
+      recordConsumer.endField("value", valueIdx);
+    }
+  }
+
+  // Return true if the logical type is a decimal with the same scale as the provided value, with enough
+  // precision to hold the value. The provided value must be a decimal.
+  private boolean compatibleDecimalType(Variant value, LogicalTypeAnnotation logicalType) {
+    if (!(logicalType instanceof LogicalTypeAnnotation.DecimalLogicalTypeAnnotation)) {
+      return false;
+    }
+    LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalType =
+        (LogicalTypeAnnotation.DecimalLogicalTypeAnnotation) logicalType;
+
+    BigDecimal decimal = value.getDecimal();
+    return decimal.scale() == decimalType.getScale() && decimal.precision() <= decimalType.getPrecision();
+  }
+
+  private boolean isTypeCompatible(Variant.Type variantType, Type typedValueField, Variant value) {
+    if (typedValueField == null) {
+      return false;
+    }
+    if (typedValueField.isPrimitive()) {
+      PrimitiveType primitiveType = typedValueField.asPrimitiveType();
+      LogicalTypeAnnotation logicalType = primitiveType.getLogicalTypeAnnotation();
+      PrimitiveType.PrimitiveTypeName primitiveTypeName = primitiveType.getPrimitiveTypeName();
+
+      switch (variantType) {
+        case BOOLEAN:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BOOLEAN;
+        case BYTE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 8;
+        case SHORT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+              && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).getBitWidth() == 16;
+        case INT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType).isSigned()
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                              .getBitWidth()
+                          == 32));
+        case LONG:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && (logicalType == null
+                  || (logicalType instanceof LogicalTypeAnnotation.IntLogicalTypeAnnotation
+                      && ((LogicalTypeAnnotation.IntLogicalTypeAnnotation) logicalType)
+                          .isSigned()));
+        case FLOAT:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.FLOAT;
+        case DOUBLE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.DOUBLE;
+        case DECIMAL4:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && compatibleDecimalType(value, logicalType);
+        case DECIMAL8:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && compatibleDecimalType(value, logicalType);
+        case DECIMAL16:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && compatibleDecimalType(value, logicalType);
+        case DATE:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT32
+              && logicalType instanceof LogicalTypeAnnotation.DateLogicalTypeAnnotation;
+        case TIME:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimeLogicalTypeAnnotation;
+        case TIMESTAMP_NTZ:
+        case TIMESTAMP_NANOS_NTZ:
+        case TIMESTAMP_TZ:
+        case TIMESTAMP_NANOS_TZ:
+          if (primitiveTypeName == PrimitiveType.PrimitiveTypeName.INT64
+              && logicalType instanceof LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimestampLogicalTypeAnnotation annotation =
+                (LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) logicalType;
+            boolean micros = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS;
+            boolean nanos = annotation.getUnit() == LogicalTypeAnnotation.TimeUnit.NANOS;
+            boolean adjustedToUTC = annotation.isAdjustedToUTC();
+            return (variantType == Variant.Type.TIMESTAMP_TZ && micros && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NTZ && micros && !adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_TZ && nanos && adjustedToUTC)
+                || (variantType == Variant.Type.TIMESTAMP_NANOS_NTZ && nanos && !adjustedToUTC);
+          } else {
+            return false;
+          }
+        case STRING:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY
+              && logicalType instanceof LogicalTypeAnnotation.StringLogicalTypeAnnotation;
+        case BINARY:
+          return primitiveTypeName == PrimitiveType.PrimitiveTypeName.BINARY && logicalType == null;
+        default:
+          return false;
+      }
+    } else if (typedValueField.getLogicalTypeAnnotation()
+        instanceof LogicalTypeAnnotation.ListLogicalTypeAnnotation) {
+      return variantType == Variant.Type.ARRAY;
+    } else {
+      return variantType == Variant.Type.OBJECT;
+    }
+  }
+
+  private void writeScalarValue(RecordConsumer recordConsumer, Variant variant, PrimitiveType type) {
+    switch (variant.getType()) {
+      case BOOLEAN:
+        recordConsumer.addBoolean(variant.getBoolean());
+        break;
+      case BYTE:
+        recordConsumer.addInteger(variant.getByte());
+        break;
+      case SHORT:
+        recordConsumer.addInteger(variant.getShort());
+        break;
+      case INT:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case LONG:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case FLOAT:
+        recordConsumer.addFloat(variant.getFloat());
+        break;
+      case DOUBLE:
+        recordConsumer.addDouble(variant.getDouble());
+        break;
+      case DECIMAL4:
+        recordConsumer.addInteger(variant.getDecimal().unscaledValue().intValue());
+        break;
+      case DECIMAL8:
+        recordConsumer.addLong(variant.getDecimal().unscaledValue().longValue());
+        break;
+      case DECIMAL16:
+        recordConsumer.addBinary(Binary.fromConstantByteArray(
+            variant.getDecimal().unscaledValue().toByteArray()));
+        break;
+      case DATE:
+        recordConsumer.addInteger(variant.getInt());
+        break;
+      case TIME:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_TZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case TIMESTAMP_NANOS_NTZ:
+        recordConsumer.addLong(variant.getLong());
+        break;
+      case STRING:
+        recordConsumer.addBinary(Binary.fromString(variant.getString()));
+        break;
+      case BINARY:
+        recordConsumer.addBinary(Binary.fromReusedByteBuffer(variant.getBinary()));
+        break;
+      default:
+        throw new IllegalArgumentException("Unsupported scalar type: " + variant.getType());
+    }
+  }
+
+  private void writeArrayValue(RecordConsumer recordConsumer, Variant variant, GroupType arrayType) {
+    Preconditions.checkArgument(
+        variant.getType() == Variant.Type.ARRAY,
+        "Cannot write variant type " + variant.getType() + " as array");
+
+    // Validate that it's a 3-level array.
+    if (arrayType.getFieldCount() != 1
+        || arrayType.getRepetition() == Type.Repetition.REPEATED
+        || arrayType.getType(0).isPrimitive()
+        || !arrayType.getFieldName(0).equals(LIST_REPEATED_NAME)) {
+      throw new IllegalArgumentException("Variant list must be a three-level list structure: " + arrayType);
+    }
+
+    // Get the element type from the array schema
+    GroupType repeatedType = arrayType.getType(0).asGroupType();
+
+    if (repeatedType.getFieldCount() != 1
+        || repeatedType.getRepetition() != Type.Repetition.REPEATED
+        || repeatedType.getType(0).isPrimitive()
+        || !repeatedType.getFieldName(0).equals(LIST_ELEMENT_NAME)) {
+      throw new IllegalArgumentException("Variant list must be a three-level list structure: " + arrayType);
+    }
+
+    GroupType elementType = repeatedType.getType(0).asGroupType();
+
+    // List field, annotated as LIST
+    recordConsumer.startGroup();
+    int numElements = variant.numArrayElements();
+    // Can only call startField if there is at least one element.
+    if (numElements > 0) {
+      recordConsumer.startField(LIST_REPEATED_NAME, 0);
+      // Write each array element
+      for (int i = 0; i < numElements; i++) {
+        // Repeated group.
+        recordConsumer.startGroup();
+        recordConsumer.startField(LIST_ELEMENT_NAME, 0);
+
+        // Element group. Can never be null for shredded Variant.
+        recordConsumer.startGroup();
+        write(elementType, variant.getElementAtIndex(i));
+        recordConsumer.endGroup();
+
+        recordConsumer.endField(LIST_ELEMENT_NAME, 0);
+        recordConsumer.endGroup();
+      }
+      recordConsumer.endField(LIST_REPEATED_NAME, 0);
+    }
+    recordConsumer.endGroup();
+  }
+
+  /**
+   * Write an object to typed_value
+   *
+   * @return the residual value that must be written to the value column, or null if all values were written
+   *         to typed_value.
+   */
+  private ByteBuffer writeObjectValue(RecordConsumer recordConsumer, Variant variant, GroupType objectType) {
+    if (variant.getType() != Variant.Type.OBJECT) {
+      throw new IllegalArgumentException("Expected object type but got: " + variant.getType());

Review Comment:
   Looks like the array case was updated but this one was not.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] GH-3223: Implement Variant parquet writer [parquet-java]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3221:
URL: https://github.com/apache/parquet-java/pull/3221#discussion_r2155720212


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantValueWriter.java:
##########
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import org.apache.parquet.io.api.Binary;
+import org.apache.parquet.io.api.RecordConsumer;
+import org.apache.parquet.schema.GroupType;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+/**
+ * Class to write Variant values to a shredded schema.
+ */
+public class VariantValueWriter {
+  private ByteBuffer metadataBuffer;
+  // We defer initializing the ImmutableMetata until it's needed. It has some construction cost, and if all
+  // object fields are shredded into typed_value, it will never be used.
+  private ImmutableMetadata metadata = null;
+  private RecordConsumer recordConsumer;
+
+  VariantValueWriter(RecordConsumer recordConsumer, ByteBuffer metadata) {
+    this.recordConsumer = recordConsumer;
+    this.metadataBuffer = metadata;
+  }
+
+  Metadata getMetadata() {
+    if (metadata == null) {
+      metadata = new ImmutableMetadata(metadataBuffer);
+    }
+    return metadata;
+  }
+
+  /**
+   * Write a Variant value to a shredded schema.
+   */
+  public static void write(RecordConsumer recordConsumer, GroupType schema, Variant value) {
+    recordConsumer.startGroup();
+    int metadataIndex = schema.getFieldIndex("metadata");
+    recordConsumer.startField("metadata", metadataIndex);
+    recordConsumer.addBinary(Binary.fromConstantByteBuffer(value.getMetadataBuffer()));
+    recordConsumer.endField("metadata", metadataIndex);
+    VariantValueWriter writer = new VariantValueWriter(recordConsumer, value.getMetadataBuffer());

Review Comment:
   I think it's fine to pass `recordConsumer` as an instance field, but right now this is mixed. `writeScalarValue` accepts a `recordConsumer` but is not a static method, so its argument has the same name as the instance field. I think this should fix that issue and either make methods static and pass in `recordConsumer` or should use the instance field. Methods that don't use instance fields should be `static`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org