[PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["gene-db (via GitHub)" <gi...@apache.org>]

gene-db opened a new pull request, #3202:
URL: https://github.com/apache/parquet-java/pull/3202

   ### Rationale for this change
   
   We need a way to build/encode Variant values. This builder allows users to create Variant values according to the [Variant binary encoding spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md).
   
   ### What changes are included in this PR?
   
   A builder class for building Variant values.
   
   ### Are these changes tested?
   
   Yes, added unit tests.
   
   ### Are there any user-facing changes?
   
   A new public class to build Variant vlues.
   
   Closes #3201


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061032993


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);

Review Comment:
   Could this header be a constant?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069540519


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObject.java:
##########
@@ -250,22 +212,61 @@ public void testMixedObject() {
     Variant value = new Variant(
         ByteBuffer.wrap(constructObject(keys, fields, true)),
         constructMetadata(true, ImmutableList.of("a", "b", "c")));
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(3, v.numObjectElements());
-      checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      VariantTestUtil.checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(1234567890, v.getFieldByKey("a").getInt());
-      checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      VariantTestUtil.checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
       Assert.assertTrue(v.getFieldByKey("b").getBoolean());
-      checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      VariantTestUtil.checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
 
       Variant nestedV = v.getFieldByKey("c");
       Assert.assertEquals(2, nestedV.numObjectElements());
-      checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
       Assert.assertEquals(
           LocalDate.parse("2025-04-17"),
           LocalDate.ofEpochDay(nestedV.getFieldByKey("a").getInt()));
-      checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+    });
+  }
+
+  @Test
+  public void testMixedObjectBuilder() {
+    VariantBuilder b = new VariantBuilder(true);
+    VariantObjectBuilder objBuilder = b.startObject();
+    b.appendObjectKey(objBuilder, "outer 1");
+    b.appendBoolean(true);
+    b.appendObjectKey(objBuilder, "outer 2");
+    b.appendLong(1234567890);
+    b.appendObjectKey(objBuilder, "outer 3");
+    {
+      // build a nested obj
+      VariantObjectBuilder nestedBuilder = b.startObject();
+      b.appendObjectKey(nestedBuilder, "nested 1");
+      {
+        // build a nested empty obj
+        VariantObjectBuilder emptyBuilder = b.startObject();
+        b.endObject(emptyBuilder);
+      }
+      b.appendObjectKey(nestedBuilder, "nested 2");
+      b.appendString("variant");
+      b.endObject(nestedBuilder);
+    }
+    b.endObject(objBuilder);
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(3, v.numObjectElements());
+      Assert.assertTrue(v.getFieldByKey("outer 1").getBoolean());
+      Assert.assertEquals(1234567890, v.getFieldByKey("outer 2").getLong());

Review Comment:
   I think this assertion needs to verify that the field was encoded as a long because `appendLong` was used. Can you add assertions for the types as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073862959


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantArrayBuilder.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestVariantArrayBuilder {
+  private static final Logger LOG = LoggerFactory.getLogger(TestVariantArrayBuilder.class);
+
+  @Test
+  public void testEmptyArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, v.numArrayElements());
+    });
+  }
+
+  @Test
+  public void testLargeArraySizeBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    for (int i = 0; i < 511; i++) {
+      a.appendInt(i);
+    }
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(511, v.numArrayElements());
+      for (int i = 0; i < 511; i++) {
+        VariantTestUtil.checkType(v.getElementAtIndex(i), VariantUtil.PRIMITIVE, Variant.Type.INT);
+        Assert.assertEquals(i, v.getElementAtIndex(i).getInt());
+      }
+    });
+  }
+
+  @Test
+  public void testMixedArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendBoolean(true);
+    VariantObjectBuilder obj = arrBuilder.startObject();
+    obj.appendKey("key");
+    obj.appendInt(321);
+    arrBuilder.endObject();
+    arrBuilder.appendLong(1234567890);
+    {
+      // build a nested array
+      VariantArrayBuilder nestedBuilder = arrBuilder.startArray();
+      {
+        // build a nested empty array
+        nestedBuilder.startArray();
+        nestedBuilder.endArray();
+      }
+      nestedBuilder.appendString("variant");
+      nestedBuilder.startObject();
+      nestedBuilder.endObject();
+      arrBuilder.endArray();
+    }
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(4, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(0).getBoolean());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(1, v.getElementAtIndex(1).numObjectElements());
+      VariantTestUtil.checkType(
+          v.getElementAtIndex(1).getFieldByKey("key"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      Assert.assertEquals(321, v.getElementAtIndex(1).getFieldByKey("key").getInt());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(3), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Variant nested = v.getElementAtIndex(3);
+      Assert.assertEquals(3, nested.numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(0), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, nested.getElementAtIndex(0).numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(1), VariantUtil.SHORT_STR, Variant.Type.STRING);
+      Assert.assertEquals("variant", nested.getElementAtIndex(1).getString());
+      VariantTestUtil.checkType(nested.getElementAtIndex(2), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, nested.getElementAtIndex(2).numObjectElements());
+    });
+  }
+
+  private void buildNested(int i, VariantArrayBuilder obj) {
+    if (i > 0) {
+      obj.appendString("str" + i);
+      buildNested(i - 1, obj.startArray());
+      obj.endArray();
+    }
+  }
+
+  @Test
+  public void testNestedBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    buildNested(1000, b.startArray());
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      Variant curr = v;
+      for (int i = 1000; i >= 0; i--) {
+        VariantTestUtil.checkType(curr, VariantUtil.ARRAY, Variant.Type.ARRAY);
+        if (i == 0) {
+          Assert.assertEquals(0, curr.numArrayElements());
+        } else {
+          Assert.assertEquals(2, curr.numArrayElements());
+          VariantTestUtil.checkType(curr.getElementAtIndex(0), VariantUtil.SHORT_STR, Variant.Type.STRING);
+          Assert.assertEquals("str" + i, curr.getElementAtIndex(0).getString());
+          curr = curr.getElementAtIndex(1);
+        }
+      }
+    });
+  }
+
+  private void testArrayOffsetSizeBuilder(String randomString) {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendString(randomString);
+    arrBuilder.appendBoolean(true);
+    arrBuilder.appendLong(1234567890);
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(3, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.STRING);
+      Assert.assertEquals(randomString, v.getElementAtIndex(0).getString());
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(1).getBoolean());
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+    });
+  }
+
+  @Test
+  public void testArrayTwoByteOffsetBuilder() {
+    // a string larger than 255 bytes to push the value offset size above 1 byte
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(300));
+  }
+
+  @Test
+  public void testArrayThreeByteOffsetBuilder() {
+    // a string larger than 65535 bytes to push the value offset size above 2 bytes
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(70_000));
+  }
+
+  @Test
+  public void testArrayFourByteOffsetBuilder() {
+    // a string larger than 16777215 bytes to push the value offset size above 3 bytes
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(16_800_000));
+  }
+
+  @Test
+  public void testMissingEndArray() {
+    VariantBuilder b = new VariantBuilder();
+    b.startArray();
+    try {
+      b.build();
+      Assert.fail("Expected Exception when calling build() without endArray()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testMissingStartArray() {

Review Comment:
   For testing exception cases, you can use `TestUtils.assertThrows` in `parquet-common`. That's a more uniform assertion that also checks the exception message.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061041198


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);
+    }
+    System.arraycopy(text, 0, writeBuffer, writePos, text.length);
+    writePos += text.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.NULL);
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(b ? VariantUtil.TRUE : VariantUtil.FALSE);
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder. The actual encoded integer type depends on the
+   * value range of the long value.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    if (l == (byte) l) {
+      checkCapacity(1 + 1);
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.INT8);
+      VariantUtil.writeLong(writeBuffer, writePos, l, 1);
+      writePos += 1;
+    } else if (l == (short) l) {
+      checkCapacity(1 + 2);
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.INT16);
+      VariantUtil.writeLong(writeBuffer, writePos, l, 2);

Review Comment:
   Wouldn't it be better to update `writePos` just once instead? This could write to `writePos + 1` and then increment by 3 on the next line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069546755


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -415,23 +574,43 @@ public void testNegativeTimestamp() {
           (byte) 0xFF,
           (byte) 0xFF
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_TZ);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_TZ);
       Assert.assertEquals(
           Instant.parse("1969-12-31T23:59:59.999999Z"), Instant.EPOCH.plus(v.getLong(), ChronoUnit.MICROS));
     });
   }
 
+  @Test
+  public void testTimestampTzBuilder() {
+    DateTimeFormatter dtf = DateTimeFormatter.ISO_DATE_TIME;
+    VariantBuilder vb = new VariantBuilder(false);
+    long micros = VariantTestUtil.microsSinceEpoch(Instant.from(dtf.parse("2024-12-16T10:23:45.321456-08:00")));

Review Comment:
   It doesn't look like converting the parsed date brings much value so I'd probably remove the conversion. It isn't worth the maintenance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2072162025


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,658 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    incrementNumValues();

Review Comment:
   As noted above, if we can do this in `onAppend` then we probably should.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061032658


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);

Review Comment:
   `text` seems like an odd name, since this is a string encoded as UTF-8 bytes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069413669


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);

Review Comment:
   This API is confusing until you know what is happening in `endObject` because the values are written, but then the data section is relocated to make room for the header (and sometimes values are copied to deduplicate keys). I would expect this to use the object builder for both key and value.
   
   It looks like the purpose of this structure is to avoid buffer allocation, so that values are always written to the `writeBuffer` directly and moved when needed, instead of staging encoded values in buffers and copying between them. These choices lead to an odd API, as well as a few passes of sorting fields and copying values around. The implementation in `endObject` is hard to understand and will be hard to maintain.
   
   There are a couple of solutions that would help.
   
   One solution is to have a separate value buffer in each `VariantObjectBuilder` and to call `appendString` or `appendDate` etc. on the object builder -- it would implement the full builder API and write field values directly to its buffer. That would have a benefit of being able to get the size of the last value written when a new field is started, and would allow better validation that the number of keys matches the number of fields. When the object is ended, the `endObject` method would simply sort the keys and copy each value between buffers. Then the object builder (and its buffer) could be reused the next time `startObject` is called by its parent builder.
   
   Another option is to keep keys and values in an in-memory map. This would have an API similar to the first option. When a value is added, you'd add it to the map using the last field key. This would require wrapping the value to keep track of its type (like Iceberg's [`PrimitiveWrapper`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/variants/PrimitiveWrapper.java)). Serialization would happen when `endObject` is called.
   
   I think both of these would be better options than the current complexity of `endObject`, and would also allow the builder API to provide better checks that it is being used correctly because the `VariantObjectBuilder` would be used for both keys and values so the parent builder would be able to fail if any `append` method is called before `endObject`. This would also avoid the need to pass the object builder back into the variant builder API.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);

Review Comment:
   This API is confusing until you know what is happening in `endObject` because the values are written, but then the data section is relocated to make room for the header (and sometimes values are copied to deduplicate keys). I would expect this to use the object builder for both key and value.
   
   It looks like the purpose of this structure is to avoid buffer allocation, so that values are always written to the `writeBuffer` directly and moved when needed, instead of staging encoded values in buffers and copying between them. These choices lead to an odd API, as well as a few passes of sorting fields and copying values around. The implementation in `endObject` is hard to understand and will be hard to maintain.
   
   There are a couple of solutions that would help.
   
   One solution is to have a separate value buffer in each `VariantObjectBuilder` and to call `appendString` or `appendDate` etc. on the object builder -- it would implement the full builder API and write field values directly to its buffer. That would have a benefit of being able to get the size of the last value written when a new field is started, and would allow better validation that the number of keys matches the number of fields. When the object is ended, the `endObject` method would simply sort the keys and copy each value between buffers. Then the object builder (and its buffer) could be reused the next time `startObject` is called by its parent builder.
   
   Another solution is to keep keys and values in an in-memory map. This would have an API similar to the first option. When a value is added, you'd add it to the map using the last field key. This would require wrapping the value to keep track of its type (like Iceberg's [`PrimitiveWrapper`](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/variants/PrimitiveWrapper.java)). Serialization would happen when `endObject` is called.
   
   I think both of these would be better options than the current complexity of `endObject`, and would also allow the builder API to provide better checks that it is being used correctly because the `VariantObjectBuilder` would be used for both keys and values so the parent builder would be able to fail if any `append` method is called before `endObject`. This would also avoid the need to pass the object builder back into the variant builder API.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069391167


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {

Review Comment:
   I find `allowDuplicateObjectKeys` confusing now that I see how it is used. This doesn't actually allow keys to have multiple values like a multi-map, it just enables a mode where the builder will replace existing keys with the last value that is set.
   
   I think that you could simplify this by always allowing duplicate object keys. There isn't much of a benefit gained by now allowing duplicate keys because duplicates cause a failure although they could be handled. And the code path to deduplicate the keys is nearly identical in the case where there are no duplicates: the first branch comparing the ID is never taken so the block that copies values to the right offset will not run.
   
   That means that the extra cost when there are no duplicate keys is the `fields.set` call that doesn't move the value. And the API would simply handle duplicate keys instead of failing with an exception.
   
   I think this should remove `allowDuplicateObjectKeys` and always handle duplicates.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["gene-db (via GitHub)" <gi...@apache.org>]

gene-db commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2071018717


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset.
+        if (fields.get(distinctPos).offset < fields.get(i).offset) {
+          fields.set(distinctPos, fields.get(i));
+        }
+      } else {
+        // Found a distinct key. Add the field to the list.
+        distinctPos++;
+        fields.set(distinctPos, fields.get(i));
+        dataSize += fields.get(i).valueSize;
+      }
+    }
+
+    if (distinctPos + 1 < fields.size()) {
+      numFields = distinctPos + 1;
+      // Resize `fields` to `size`.
+      fields.subList(numFields, fields.size()).clear();
+    }
+
+    boolean largeSize = numFields > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + numFields * idSize + (numFields + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Write the header byte and size entry.
+    writeBuffer[writePos] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numFields, sizeBytes);
+
+    int idStart = writePos + 1 + sizeBytes;
+    int offsetStart = idStart + numFields * idSize;
+    int fieldDataStart = offsetStart + (numFields + 1) * offsetSize;
+    int currOffset = 0;
+
+    // Loop over all fields and write the key id, offset, and data to the appropriate offsets
+    for (int i = 0; i < numFields; ++i) {
+      FieldEntry field = fields.get(i);
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, field.id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, currOffset, offsetSize);
+      System.arraycopy(
+          objectBuilder.writeBuffer, field.offset, writeBuffer, fieldDataStart + currOffset, field.valueSize);
+      currOffset += field.valueSize;
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numFields * offsetSize, dataSize, offsetSize);
+    writePos += headerSize + dataSize;
+    numValues++;
+    objectBuilder = null;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder is used to
+   * append values ot the array. startArray() must be called before endArray(). No append*() methods
+   * can be called in between startArray() and endArray().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.appendString("value1");
+   * arrayBuilder.appendString("value2");
+   * builder.endArray();
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endArray() first.");
+    }
+    arrayBuilder = new VariantArrayBuilder(this);
+    return arrayBuilder;
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   */
+  public void endArray() {
+    if (arrayBuilder == null) {
+      throw new IllegalStateException("Cannot call endArray() without calling startArray() first.");
+    }
+    ArrayList<Integer> offsets = arrayBuilder.validateAndGetOffsets();
+    int numElements = offsets.size();
+    int dataSize = arrayBuilder.writePos;
+    boolean largeSize = numElements > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (numElements + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Copy all the element data to the write buffer.
+    System.arraycopy(arrayBuilder.writeBuffer, 0, writeBuffer, writePos + headerSize, dataSize);
+
+    writeBuffer[writePos] = VariantUtil.arrayHeader(largeSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numElements, sizeBytes);
+
+    int offsetStart = writePos + 1 + sizeBytes;
+    for (int i = 0; i < numElements; ++i) {
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, offsets.get(i), offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numElements * offsetSize, dataSize, offsetSize);
+    writePos += headerSize + dataSize;
+    numValues++;
+    arrayBuilder = null;
+  }
+
+  protected void onAppend() {
+    checkAppendWhileNested();
+    if (numValues > 0) {

Review Comment:
   Yes, updated.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset.
+        if (fields.get(distinctPos).offset < fields.get(i).offset) {
+          fields.set(distinctPos, fields.get(i));
+        }
+      } else {
+        // Found a distinct key. Add the field to the list.
+        distinctPos++;
+        fields.set(distinctPos, fields.get(i));
+        dataSize += fields.get(i).valueSize;
+      }
+    }
+
+    if (distinctPos + 1 < fields.size()) {
+      numFields = distinctPos + 1;
+      // Resize `fields` to `size`.
+      fields.subList(numFields, fields.size()).clear();
+    }
+
+    boolean largeSize = numFields > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + numFields * idSize + (numFields + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Write the header byte and size entry.
+    writeBuffer[writePos] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numFields, sizeBytes);
+
+    int idStart = writePos + 1 + sizeBytes;
+    int offsetStart = idStart + numFields * idSize;
+    int fieldDataStart = offsetStart + (numFields + 1) * offsetSize;
+    int currOffset = 0;
+
+    // Loop over all fields and write the key id, offset, and data to the appropriate offsets
+    for (int i = 0; i < numFields; ++i) {
+      FieldEntry field = fields.get(i);
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, field.id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, currOffset, offsetSize);
+      System.arraycopy(
+          objectBuilder.writeBuffer, field.offset, writeBuffer, fieldDataStart + currOffset, field.valueSize);
+      currOffset += field.valueSize;
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numFields * offsetSize, dataSize, offsetSize);
+    writePos += headerSize + dataSize;
+    numValues++;
+    objectBuilder = null;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder is used to
+   * append values ot the array. startArray() must be called before endArray(). No append*() methods
+   * can be called in between startArray() and endArray().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.appendString("value1");
+   * arrayBuilder.appendString("value2");
+   * builder.endArray();
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endArray() first.");
+    }
+    arrayBuilder = new VariantArrayBuilder(this);
+    return arrayBuilder;
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   */
+  public void endArray() {
+    if (arrayBuilder == null) {
+      throw new IllegalStateException("Cannot call endArray() without calling startArray() first.");
+    }
+    ArrayList<Integer> offsets = arrayBuilder.validateAndGetOffsets();
+    int numElements = offsets.size();
+    int dataSize = arrayBuilder.writePos;
+    boolean largeSize = numElements > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (numElements + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Copy all the element data to the write buffer.
+    System.arraycopy(arrayBuilder.writeBuffer, 0, writeBuffer, writePos + headerSize, dataSize);

Review Comment:
   Renamed.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset.

Review Comment:
   Added.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;

Review Comment:
   Removed this.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);

Review Comment:
   Added the `this.` prefix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061012912


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;

Review Comment:
   The variable name here doesn't quite make sense to me because this isn't the size of a string, it is the size of all strings in the data/content section of the dictionary. What about `totalDataSize` or `totalContentSize`?
   
   Looks like there are other variable names that refer to the dictionary content bytes as the "string" that could be clarified as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061005658


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.

Review Comment:
   Duplicate object keys? This could be more clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2072156036


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantArrayBuilder.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant arrays, used by VariantBuilder.
+ */
+public class VariantArrayBuilder extends VariantBuilder {
+  /** The parent VariantBuilder. */
+  private final VariantBuilder parent;
+  /** The offsets of the elements in this array. */
+  private final ArrayList<Integer> offsets;
+  /** The number of values appended to this array. */
+  protected long numValues = 0;
+
+  VariantArrayBuilder(VariantBuilder parent) {
+    this.parent = parent;
+    this.offsets = new ArrayList<>();
+  }
+
+  /**
+   * @return the list of element offsets in this array
+   */
+  ArrayList<Integer> validateAndGetOffsets() {
+    if (offsets.size() != numValues) {
+      throw new IllegalStateException(String.format(
+          "Number of offsets (%d) do not match the number of values (%d).", offsets.size(), numValues));
+    }
+    return offsets;
+  }
+
+  @Override
+  protected void onAppend() {
+    checkAppendWhileNested();
+    offsets.add(writePos);
+  }
+
+  @Override
+  protected void onStartNested() {
+    checkMultipleNested();
+    offsets.add(writePos);
+  }
+
+  @Override
+  protected void incrementNumValues() {

Review Comment:
   Why does this require a protected method rather than putting this in `onAppend`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2070758144


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset.
+        if (fields.get(distinctPos).offset < fields.get(i).offset) {
+          fields.set(distinctPos, fields.get(i));
+        }
+      } else {
+        // Found a distinct key. Add the field to the list.
+        distinctPos++;
+        fields.set(distinctPos, fields.get(i));
+        dataSize += fields.get(i).valueSize;
+      }
+    }
+
+    if (distinctPos + 1 < fields.size()) {
+      numFields = distinctPos + 1;
+      // Resize `fields` to `size`.
+      fields.subList(numFields, fields.size()).clear();
+    }
+
+    boolean largeSize = numFields > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + numFields * idSize + (numFields + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Write the header byte and size entry.
+    writeBuffer[writePos] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numFields, sizeBytes);
+
+    int idStart = writePos + 1 + sizeBytes;
+    int offsetStart = idStart + numFields * idSize;
+    int fieldDataStart = offsetStart + (numFields + 1) * offsetSize;
+    int currOffset = 0;
+
+    // Loop over all fields and write the key id, offset, and data to the appropriate offsets
+    for (int i = 0; i < numFields; ++i) {
+      FieldEntry field = fields.get(i);
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, field.id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, currOffset, offsetSize);
+      System.arraycopy(
+          objectBuilder.writeBuffer, field.offset, writeBuffer, fieldDataStart + currOffset, field.valueSize);
+      currOffset += field.valueSize;
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numFields * offsetSize, dataSize, offsetSize);
+    writePos += headerSize + dataSize;
+    numValues++;
+    objectBuilder = null;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder is used to
+   * append values ot the array. startArray() must be called before endArray(). No append*() methods
+   * can be called in between startArray() and endArray().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.appendString("value1");
+   * arrayBuilder.appendString("value2");
+   * builder.endArray();
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endArray() first.");
+    }
+    arrayBuilder = new VariantArrayBuilder(this);
+    return arrayBuilder;
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   */
+  public void endArray() {
+    if (arrayBuilder == null) {
+      throw new IllegalStateException("Cannot call endArray() without calling startArray() first.");
+    }
+    ArrayList<Integer> offsets = arrayBuilder.validateAndGetOffsets();
+    int numElements = offsets.size();
+    int dataSize = arrayBuilder.writePos;
+    boolean largeSize = numElements > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (numElements + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Copy all the element data to the write buffer.
+    System.arraycopy(arrayBuilder.writeBuffer, 0, writeBuffer, writePos + headerSize, dataSize);
+
+    writeBuffer[writePos] = VariantUtil.arrayHeader(largeSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numElements, sizeBytes);
+
+    int offsetStart = writePos + 1 + sizeBytes;
+    for (int i = 0; i < numElements; ++i) {
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, offsets.get(i), offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numElements * offsetSize, dataSize, offsetSize);
+    writePos += headerSize + dataSize;
+    numValues++;
+    arrayBuilder = null;
+  }
+
+  protected void onAppend() {
+    checkAppendWhileNested();
+    if (numValues > 0) {

Review Comment:
   This could be checked using `writePos` instead, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069414671


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {
+      int distinctPos = 0;
+      // Maintain a list of distinct keys in-place.
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        if (fields.get(i).id == fields.get(i - 1).id) {
+          // Found a duplicate key. Keep the field with the greater offset.
+          if (fields.get(distinctPos).offset < fields.get(i).offset) {
+            fields.set(distinctPos, fields.get(distinctPos).withNewOffset(fields.get(i).offset));
+          }
+        } else {
+          // Found a distinct key. Add the field to the list.
+          ++distinctPos;
+          fields.set(distinctPos, fields.get(i));
+        }
+      }
+      if (distinctPos + 1 < fields.size()) {
+        size = distinctPos + 1;
+        // Resize `fields` to `size`.
+        fields.subList(size, fields.size()).clear();
+        // Sort the fields by offsets so that we can move the value data of each field to the new
+        // offset without overwriting the fields after it.
+        fields.sort(Comparator.comparingInt(f -> f.offset));
+        int currentOffset = 0;
+        for (int i = 0; i < size; ++i) {
+          int oldOffset = fields.get(i).offset;
+          int fieldSize =
+              VariantUtil.valueSize(ByteBuffer.wrap(writeBuffer), objBuilder.startPos() + oldOffset);
+          System.arraycopy(
+              writeBuffer,
+              objBuilder.startPos() + oldOffset,
+              writeBuffer,
+              objBuilder.startPos() + currentOffset,
+              fieldSize);
+          fields.set(i, fields.get(i).withNewOffset(currentOffset));
+          currentOffset += fieldSize;
+        }
+        writePos = objBuilder.startPos() + currentOffset;
+        // Change back to the sort order by field keys, required by the Variant specification.
+        Collections.sort(fields);
+      }
+    } else {
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        String key = fields.get(i).key;
+        if (key.equals(fields.get(i - 1).key)) {
+          throw new IllegalStateException("Failed to build Variant because of duplicate object key: " + key);
+        }
+      }
+    }
+    int dataSize = writePos - objBuilder.startPos();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + size * idSize + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the object header section.
+    System.arraycopy(writeBuffer, objBuilder.startPos(), writeBuffer, objBuilder.startPos() + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[objBuilder.startPos()] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, objBuilder.startPos() + 1, size, sizeBytes);
+    int idStart = objBuilder.startPos() + 1 + sizeBytes;
+    int offsetStart = idStart + size * idSize;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, fields.get(i).id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, fields.get(i).offset, offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder must be
+   * used for future calls to startArrayElement() and endArray(). To add an element to the array,
+   * call startArrayElement() and then append the value. endArray() must be called to finish writing
+   * the Variant array.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.startArrayElement(arrayBuilder);
+   * builder.appendString("value1");
+   * arrayBuilder.endArray(arrayBuilder);
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    return new VariantArrayBuilder(this);
+  }
+
+  /**
+   * Starts appending an element to the array. This method must be called before appending the
+   * corresponding value.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   */
+  public void startArrayElement(VariantArrayBuilder arrayBuilder) {
+    arrayBuilder.startElement();
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endArray(VariantArrayBuilder arrayBuilder) {

Review Comment:
   I think that once the object builder is updated, the same design should be used to update the array builder.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069421805


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {
+      int distinctPos = 0;
+      // Maintain a list of distinct keys in-place.
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        if (fields.get(i).id == fields.get(i - 1).id) {
+          // Found a duplicate key. Keep the field with the greater offset.
+          if (fields.get(distinctPos).offset < fields.get(i).offset) {
+            fields.set(distinctPos, fields.get(distinctPos).withNewOffset(fields.get(i).offset));
+          }
+        } else {
+          // Found a distinct key. Add the field to the list.
+          ++distinctPos;
+          fields.set(distinctPos, fields.get(i));
+        }
+      }
+      if (distinctPos + 1 < fields.size()) {
+        size = distinctPos + 1;
+        // Resize `fields` to `size`.
+        fields.subList(size, fields.size()).clear();
+        // Sort the fields by offsets so that we can move the value data of each field to the new
+        // offset without overwriting the fields after it.
+        fields.sort(Comparator.comparingInt(f -> f.offset));
+        int currentOffset = 0;
+        for (int i = 0; i < size; ++i) {
+          int oldOffset = fields.get(i).offset;
+          int fieldSize =
+              VariantUtil.valueSize(ByteBuffer.wrap(writeBuffer), objBuilder.startPos() + oldOffset);
+          System.arraycopy(
+              writeBuffer,
+              objBuilder.startPos() + oldOffset,
+              writeBuffer,
+              objBuilder.startPos() + currentOffset,
+              fieldSize);
+          fields.set(i, fields.get(i).withNewOffset(currentOffset));
+          currentOffset += fieldSize;
+        }
+        writePos = objBuilder.startPos() + currentOffset;
+        // Change back to the sort order by field keys, required by the Variant specification.
+        Collections.sort(fields);
+      }
+    } else {
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        String key = fields.get(i).key;
+        if (key.equals(fields.get(i - 1).key)) {
+          throw new IllegalStateException("Failed to build Variant because of duplicate object key: " + key);
+        }
+      }
+    }
+    int dataSize = writePos - objBuilder.startPos();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + size * idSize + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the object header section.
+    System.arraycopy(writeBuffer, objBuilder.startPos(), writeBuffer, objBuilder.startPos() + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[objBuilder.startPos()] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, objBuilder.startPos() + 1, size, sizeBytes);
+    int idStart = objBuilder.startPos() + 1 + sizeBytes;
+    int offsetStart = idStart + size * idSize;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, fields.get(i).id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, fields.get(i).offset, offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder must be
+   * used for future calls to startArrayElement() and endArray(). To add an element to the array,
+   * call startArrayElement() and then append the value. endArray() must be called to finish writing
+   * the Variant array.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.startArrayElement(arrayBuilder);
+   * builder.appendString("value1");
+   * arrayBuilder.endArray(arrayBuilder);
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    return new VariantArrayBuilder(this);
+  }
+
+  /**
+   * Starts appending an element to the array. This method must be called before appending the
+   * corresponding value.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   */
+  public void startArrayElement(VariantArrayBuilder arrayBuilder) {
+    arrayBuilder.startElement();
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endArray(VariantArrayBuilder arrayBuilder) {
+    int start = arrayBuilder.startPos();
+    int dataSize = writePos - start;
+    int size = arrayBuilder.offsets().size();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the header section.
+    System.arraycopy(writeBuffer, start, writeBuffer, start + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[start] = VariantUtil.arrayHeader(largeSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, start + 1, size, sizeBytes);
+    int offsetStart = start + 1 + sizeBytes;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(
+          writeBuffer,
+          offsetStart + i * offsetSize,
+          arrayBuilder.offsets().get(i),
+          offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Adds a key to the Variant dictionary. If the key already exists, the dictionary is unmodified.
+   * @param key the key to add
+   * @return the id of the key
+   */
+  int addKey(String key) {
+    return dictionary.computeIfAbsent(key, newKey -> {
+      int id = dictionaryKeys.size();
+      dictionaryKeys.add(newKey.getBytes(StandardCharsets.UTF_8));
+      return id;
+    });
+  }
+
+  /**
+   * @return the current write position of the variant builder
+   */
+  int writePos() {
+    return writePos;
+  }
+
+  /**
+   * Class to store the information of a Variant object field. We need to collect all fields of
+   * an object, sort them by their keys, and build the Variant object in sorted order.
+   */
+  static final class FieldEntry implements Comparable<FieldEntry> {
+    final String key;
+    final int id;
+    final int offset;
+
+    FieldEntry(String key, int id, int offset) {
+      this.key = key;
+      this.id = id;
+      this.offset = offset;
+    }
+
+    FieldEntry withNewOffset(int newOffset) {
+      return new FieldEntry(key, id, newOffset);
+    }
+
+    @Override
+    public int compareTo(FieldEntry other) {
+      return key.compareTo(other.key);
+    }
+  }
+
+  private void checkCapacity(int additionalBytes) {
+    int requiredBytes = writePos + additionalBytes;
+    if (requiredBytes > writeBuffer.length) {
+      // Allocate a new buffer with a capacity of the next power of 2 of `requiredBytes`.
+      int newCapacity = Integer.highestOneBit(requiredBytes);
+      newCapacity = newCapacity < requiredBytes ? newCapacity * 2 : newCapacity;
+      byte[] newValue = new byte[newCapacity];
+      System.arraycopy(writeBuffer, 0, newValue, 0, writePos);
+      writeBuffer = newValue;
+    }
+  }
+
+  private int getMinIntegerSize(int value) {
+    assert value >= 0;
+    if (value <= VariantUtil.U8_MAX) {
+      return VariantUtil.U8_SIZE;
+    }
+    if (value <= VariantUtil.U16_MAX) {
+      return VariantUtil.U16_SIZE;
+    }
+    if (value <= VariantUtil.U24_MAX) {
+      return VariantUtil.U24_SIZE;
+    }
+    return VariantUtil.U32_SIZE;
+  }
+
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  private byte[] writeBuffer = new byte[128];
+
+  private int writePos = 0;

Review Comment:
   As with the other builders, this would be better at the top.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069427878


##########
parquet-variant/src/test/java/org/apache/parquet/variant/VariantTestUtil.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.security.SecureRandom;
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import org.junit.Assert;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VariantTestUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(VariantTestUtil.class);
+  private static final String RANDOM_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
+
+  /** Random number generator for generating random strings */
+  private static SecureRandom random = new SecureRandom(new byte[] {1, 2, 3, 4, 5});
+
+  static final ByteBuffer EMPTY_METADATA = ByteBuffer.wrap(new byte[] {0b1});
+
+  static void checkType(Variant v, int expectedBasicType, Variant.Type expectedType) {
+    Assert.assertEquals(expectedBasicType, v.value.get(v.value.position()) & VariantUtil.BASIC_TYPE_MASK);
+    Assert.assertEquals(expectedType, v.getType());
+  }
+
+  static long microsSinceEpoch(Instant instant) {

Review Comment:
   Why is this interacting with `Instant`? There is no need to have a conversion here because the write methods are called with `long` values not `Instant`. Tests only need to validate that the value passed in matches the value read back and that the type is preserved. Conversion from a date/time representation is the responsibility of engines, not Parquet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069537132


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObject.java:
##########
@@ -250,22 +212,61 @@ public void testMixedObject() {
     Variant value = new Variant(
         ByteBuffer.wrap(constructObject(keys, fields, true)),
         constructMetadata(true, ImmutableList.of("a", "b", "c")));
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(3, v.numObjectElements());
-      checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      VariantTestUtil.checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(1234567890, v.getFieldByKey("a").getInt());
-      checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      VariantTestUtil.checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
       Assert.assertTrue(v.getFieldByKey("b").getBoolean());
-      checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      VariantTestUtil.checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
 
       Variant nestedV = v.getFieldByKey("c");
       Assert.assertEquals(2, nestedV.numObjectElements());
-      checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
       Assert.assertEquals(
           LocalDate.parse("2025-04-17"),
           LocalDate.ofEpochDay(nestedV.getFieldByKey("a").getInt()));
-      checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+    });
+  }
+
+  @Test
+  public void testMixedObjectBuilder() {

Review Comment:
   I would prefer to have builder tests in a `TestVariantBuilder` or `TestVariantObjectBuilder` suite to keep them separate from the decoding tests. These tests validate the `Variant` decoding and that is used to validate the encoding/builder.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073868996


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantArrayBuilder.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestVariantArrayBuilder {
+  private static final Logger LOG = LoggerFactory.getLogger(TestVariantArrayBuilder.class);
+
+  @Test
+  public void testEmptyArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, v.numArrayElements());
+    });
+  }
+
+  @Test
+  public void testLargeArraySizeBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    for (int i = 0; i < 511; i++) {
+      a.appendInt(i);
+    }
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(511, v.numArrayElements());
+      for (int i = 0; i < 511; i++) {
+        VariantTestUtil.checkType(v.getElementAtIndex(i), VariantUtil.PRIMITIVE, Variant.Type.INT);
+        Assert.assertEquals(i, v.getElementAtIndex(i).getInt());
+      }
+    });
+  }
+
+  @Test
+  public void testMixedArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendBoolean(true);
+    VariantObjectBuilder obj = arrBuilder.startObject();
+    obj.appendKey("key");
+    obj.appendInt(321);
+    arrBuilder.endObject();
+    arrBuilder.appendLong(1234567890);
+    {
+      // build a nested array
+      VariantArrayBuilder nestedBuilder = arrBuilder.startArray();
+      {
+        // build a nested empty array
+        nestedBuilder.startArray();
+        nestedBuilder.endArray();
+      }
+      nestedBuilder.appendString("variant");
+      nestedBuilder.startObject();
+      nestedBuilder.endObject();
+      arrBuilder.endArray();
+    }
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(4, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(0).getBoolean());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(1, v.getElementAtIndex(1).numObjectElements());
+      VariantTestUtil.checkType(
+          v.getElementAtIndex(1).getFieldByKey("key"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      Assert.assertEquals(321, v.getElementAtIndex(1).getFieldByKey("key").getInt());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(3), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Variant nested = v.getElementAtIndex(3);
+      Assert.assertEquals(3, nested.numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(0), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, nested.getElementAtIndex(0).numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(1), VariantUtil.SHORT_STR, Variant.Type.STRING);
+      Assert.assertEquals("variant", nested.getElementAtIndex(1).getString());
+      VariantTestUtil.checkType(nested.getElementAtIndex(2), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, nested.getElementAtIndex(2).numObjectElements());
+    });
+  }
+
+  private void buildNested(int i, VariantArrayBuilder obj) {
+    if (i > 0) {
+      obj.appendString("str" + i);
+      buildNested(i - 1, obj.startArray());
+      obj.endArray();
+    }
+  }
+
+  @Test
+  public void testNestedBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    buildNested(1000, b.startArray());
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      Variant curr = v;
+      for (int i = 1000; i >= 0; i--) {
+        VariantTestUtil.checkType(curr, VariantUtil.ARRAY, Variant.Type.ARRAY);
+        if (i == 0) {
+          Assert.assertEquals(0, curr.numArrayElements());
+        } else {
+          Assert.assertEquals(2, curr.numArrayElements());
+          VariantTestUtil.checkType(curr.getElementAtIndex(0), VariantUtil.SHORT_STR, Variant.Type.STRING);
+          Assert.assertEquals("str" + i, curr.getElementAtIndex(0).getString());
+          curr = curr.getElementAtIndex(1);
+        }
+      }
+    });
+  }
+
+  private void testArrayOffsetSizeBuilder(String randomString) {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendString(randomString);
+    arrBuilder.appendBoolean(true);
+    arrBuilder.appendLong(1234567890);
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(3, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.STRING);
+      Assert.assertEquals(randomString, v.getElementAtIndex(0).getString());
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(1).getBoolean());
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+    });
+  }
+
+  @Test
+  public void testArrayTwoByteOffsetBuilder() {
+    // a string larger than 255 bytes to push the value offset size above 1 byte
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(300));
+  }
+
+  @Test
+  public void testArrayThreeByteOffsetBuilder() {
+    // a string larger than 65535 bytes to push the value offset size above 2 bytes
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(70_000));
+  }
+
+  @Test
+  public void testArrayFourByteOffsetBuilder() {
+    // a string larger than 16777215 bytes to push the value offset size above 3 bytes
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(16_800_000));
+  }
+
+  @Test
+  public void testMissingEndArray() {
+    VariantBuilder b = new VariantBuilder();
+    b.startArray();
+    try {
+      b.build();
+      Assert.fail("Expected Exception when calling build() without endArray()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testMissingStartArray() {
+    VariantBuilder b = new VariantBuilder();
+    try {
+      b.endArray();
+      Assert.fail("Expected Exception when calling endArray() without startArray()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testInvalidAppendDuringArray() {
+    VariantBuilder b = new VariantBuilder();
+    b.startArray();
+    try {
+      b.appendInt(1);
+      Assert.fail("Expected Exception when calling append() before endArray()");

Review Comment:
   I don't think this exception message is helpful because the API is a bit confusing. Some similar builders (like Jackson) would change the state of the current builder, rather than forcing the caller to use a new builder that is returned. It's okay to make this choice about how the builders are related, but the error message needs to tell the caller what to do. In this case, the caller should be informed that `append` methods must be called on the builder returned by `startArray` and `endArray` is called on the original builder when those are complete.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073888933


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);

Review Comment:
   If we aren't going to support reuse right away, then there is no reason to copy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073884093


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObjectBuilder.java:
##########
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import org.junit.Assert;
+import org.junit.Ignore;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestVariantObjectBuilder {
+  private static final Logger LOG = LoggerFactory.getLogger(TestVariantObjectBuilder.class);
+
+  @Test
+  public void testEmptyObjectBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    b.startObject();
+    b.endObject();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, v.numObjectElements());
+    });
+  }
+
+  @Test
+  public void testLargeObjectBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder o = b.startObject();
+    for (int i = 0; i < 1234; i++) {
+      o.appendKey("a" + i);
+      o.appendLong(i);
+    }
+    b.endObject();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(1234, v.numObjectElements());
+      for (int i = 0; i < 1234; i++) {
+        VariantTestUtil.checkType(v.getFieldByKey("a" + i), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+        Assert.assertEquals(i, v.getFieldByKey("a" + i).getLong());
+      }
+    });
+  }
+
+  @Test
+  public void testMixedObjectBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder objBuilder = b.startObject();
+    objBuilder.appendKey("outer 2");
+    objBuilder.appendLong(1234567890);
+    objBuilder.appendKey("outer 1");
+    objBuilder.appendBoolean(true);
+    objBuilder.appendKey("outer 4");
+    objBuilder.startArray();
+    objBuilder.endArray();
+    objBuilder.appendKey("outer 3");
+    {
+      // build a nested obj
+      VariantObjectBuilder nestedBuilder = objBuilder.startObject();
+      nestedBuilder.appendKey("nested 3");
+      VariantArrayBuilder arr = nestedBuilder.startArray();
+      arr.appendInt(321);
+      nestedBuilder.endArray();
+      nestedBuilder.appendKey("nested 1");
+      {
+        // build a nested empty obj
+        nestedBuilder.startObject();
+        nestedBuilder.endObject();
+      }
+      nestedBuilder.appendKey("nested 2");
+      nestedBuilder.appendString("variant");
+      objBuilder.endObject();
+    }
+    b.endObject();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(4, v.numObjectElements());
+      VariantTestUtil.checkType(v.getFieldByKey("outer 1"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getFieldByKey("outer 1").getBoolean());
+      VariantTestUtil.checkType(v.getFieldByKey("outer 2"), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getFieldByKey("outer 2").getLong());
+      VariantTestUtil.checkType(v.getFieldByKey("outer 3"), VariantUtil.OBJECT, Variant.Type.OBJECT);
+
+      Variant nested = v.getFieldByKey("outer 3");
+      Assert.assertEquals(3, nested.numObjectElements());
+      VariantTestUtil.checkType(nested.getFieldByKey("nested 1"), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, nested.getFieldByKey("nested 1").numObjectElements());
+      VariantTestUtil.checkType(nested.getFieldByKey("nested 2"), VariantUtil.SHORT_STR, Variant.Type.STRING);
+      Assert.assertEquals("variant", nested.getFieldByKey("nested 2").getString());
+      VariantTestUtil.checkType(nested.getFieldByKey("nested 3"), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(1, nested.getFieldByKey("nested 3").numArrayElements());
+      VariantTestUtil.checkType(
+          nested.getFieldByKey("nested 3").getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      Assert.assertEquals(
+          321, nested.getFieldByKey("nested 3").getElementAtIndex(0).getInt());
+
+      VariantTestUtil.checkType(v.getFieldByKey("outer 4"), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, v.getFieldByKey("outer 4").numArrayElements());
+    });
+  }
+
+  private void buildNested(int i, VariantObjectBuilder obj) {
+    if (i > 0) {
+      obj.appendKey("key" + i);
+      obj.appendString("str" + i);
+      obj.appendKey("duplicate");
+      buildNested(i - 1, obj.startObject());
+      obj.endObject();
+    }
+  }
+
+  @Test
+  public void testNestedBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    buildNested(1000, b.startObject());
+    b.endObject();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      Variant curr = v;
+      for (int i = 1000; i >= 0; i--) {
+        VariantTestUtil.checkType(curr, VariantUtil.OBJECT, Variant.Type.OBJECT);
+        if (i == 0) {
+          Assert.assertEquals(0, curr.numObjectElements());
+        } else {
+          Assert.assertEquals(2, curr.numObjectElements());
+          VariantTestUtil.checkType(
+              curr.getFieldByKey("key" + i), VariantUtil.SHORT_STR, Variant.Type.STRING);
+          Assert.assertEquals("str" + i, curr.getFieldByKey("key" + i).getString());
+          curr = curr.getFieldByKey("duplicate");
+        }
+      }
+    });
+  }
+
+  private void testObjectOffsetSizeBuilder(String randomString) {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder objBuilder = b.startObject();
+    objBuilder.appendKey("key1");
+    objBuilder.appendString(randomString);
+    objBuilder.appendKey("key2");
+    objBuilder.appendBoolean(true);
+    objBuilder.appendKey("key3");
+    objBuilder.appendLong(1234567890);
+    b.endObject();
+
+    Variant v = b.build();
+    VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    Assert.assertEquals(3, v.numObjectElements());
+    VariantTestUtil.checkType(v.getFieldByKey("key1"), VariantUtil.PRIMITIVE, Variant.Type.STRING);
+    Assert.assertEquals(randomString, v.getFieldByKey("key1").getString());
+    VariantTestUtil.checkType(v.getFieldByKey("key2"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+    Assert.assertTrue(v.getFieldByKey("key2").getBoolean());
+    VariantTestUtil.checkType(v.getFieldByKey("key3"), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+    Assert.assertEquals(1234567890, v.getFieldByKey("key3").getLong());
+  }
+
+  @Test
+  public void testObjectTwoByteOffsetBuilder() {
+    // a string larger than 255 bytes to push the offset size above 1 byte
+    testObjectOffsetSizeBuilder(VariantTestUtil.randomString(300));
+  }
+
+  @Test
+  public void testObjectThreeByteOffsetBuilder() {
+    // a string larger than 65535 bytes to push the offset size above 2 bytes
+    testObjectOffsetSizeBuilder(VariantTestUtil.randomString(70_000));
+  }
+
+  @Test
+  public void testObjectFourByteOffsetBuilder() {
+    // a string larger than 16777215 bytes to push the offset size above 3 bytes
+    testObjectOffsetSizeBuilder(VariantTestUtil.randomString(16_800_000));
+  }
+
+  private void testObjectFieldIdSizeBuilder(int numKeys) {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder objBuilder = b.startObject();
+    for (int i = 0; i < numKeys; i++) {
+      objBuilder.appendKey("k" + i);
+      objBuilder.appendLong(i);
+    }
+    b.endObject();
+
+    Variant v = b.build();
+    VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    Assert.assertEquals(numKeys, v.numObjectElements());
+    // Only check a few keys, to avoid slowing down the test
+    VariantTestUtil.checkType(v.getFieldByKey("k" + 0), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+    Assert.assertEquals(0, v.getFieldByKey("k" + 0).getLong());
+    VariantTestUtil.checkType(v.getFieldByKey("k" + (numKeys - 1)), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+    Assert.assertEquals(numKeys - 1, v.getFieldByKey("k" + (numKeys - 1)).getLong());
+  }
+
+  @Test
+  public void testObjectTwoByteFieldIdBuilder() {
+    // need more than 255 dictionary entries to push field id size above 1 byte
+    testObjectFieldIdSizeBuilder(300);
+  }
+
+  @Test
+  public void testObjectThreeByteFieldIdBuilder() {
+    // need more than 65535 dictionary entries to push field id size above 2 bytes
+    testObjectFieldIdSizeBuilder(70_000);
+  }
+
+  @Test
+  @Ignore("Test uses too much memory")
+  public void testObjectFourByteFieldIdBuilder() {
+    // need more than 16777215 dictionary entries to push field id size above 3 bytes
+    testObjectFieldIdSizeBuilder(16_800_000);
+  }
+
+  @Test
+  public void testDuplicateKeys() {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder objBuilder = b.startObject();
+    objBuilder.appendKey("duplicate");
+    objBuilder.appendLong(0);
+    objBuilder.appendKey("duplicate");
+    objBuilder.appendLong(1);
+    b.endObject();
+    Variant v = b.build();
+    Assert.assertEquals(1, v.numObjectElements());
+    VariantTestUtil.checkType(v.getFieldByKey("duplicate"), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+    Assert.assertEquals(1, v.getFieldByKey("duplicate").getLong());
+  }
+
+  @Test
+  public void testSortingKeys() {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder objBuilder = b.startObject();
+    objBuilder.appendKey("1");
+    objBuilder.appendString("1");
+    objBuilder.appendKey("0");
+    objBuilder.appendString("");
+    objBuilder.appendKey("3");
+    objBuilder.appendString("333");
+    objBuilder.appendKey("2");
+    objBuilder.appendString("22");
+    b.endObject();
+    Variant v = b.build();
+    Assert.assertEquals(4, v.numObjectElements());
+    VariantTestUtil.checkType(v.getFieldByKey("0"), VariantUtil.SHORT_STR, Variant.Type.STRING);
+    Assert.assertEquals("", v.getFieldByKey("0").getString());
+    VariantTestUtil.checkType(v.getFieldByKey("1"), VariantUtil.SHORT_STR, Variant.Type.STRING);
+    Assert.assertEquals("1", v.getFieldByKey("1").getString());
+    VariantTestUtil.checkType(v.getFieldByKey("2"), VariantUtil.SHORT_STR, Variant.Type.STRING);
+    Assert.assertEquals("22", v.getFieldByKey("2").getString());
+    VariantTestUtil.checkType(v.getFieldByKey("3"), VariantUtil.SHORT_STR, Variant.Type.STRING);
+    Assert.assertEquals("333", v.getFieldByKey("3").getString());
+  }
+
+  @Test
+  public void testMissingEndObject() {
+    VariantBuilder b = new VariantBuilder();
+    b.startObject();
+    try {
+      b.build();
+      Assert.fail("Expected Exception when calling build() without endObject()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testMissingStartObject() {
+    VariantBuilder b = new VariantBuilder();
+    try {
+      b.endObject();
+      Assert.fail("Expected Exception when calling endObject() without startObject()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testMissingValue() {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder obj = b.startObject();
+    obj.appendKey("a");
+    try {
+      b.endObject();
+      Assert.fail("Expected Exception when calling endObject() with mismatched keys and values");
+    } catch (Exception e) {
+      // expected
+    }
+
+    obj.appendInt(1);
+    obj.appendKey("b");
+    try {
+      b.endObject();
+      Assert.fail("Expected Exception when calling endObject() with mismatched keys and values");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testInvalidAppendDuringObjectAppend() {
+    VariantBuilder b = new VariantBuilder();
+    b.startObject();
+    try {
+      b.appendInt(1);
+      Assert.fail("Expected Exception when calling append() before endObject()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testMultipleAppendKey() {
+    VariantBuilder b = new VariantBuilder();
+    VariantObjectBuilder obj = b.startObject();
+    obj.appendKey("a");
+    try {
+      obj.appendKey("a");

Review Comment:
   Can you use a different key to ensure that it isn't a duplicate key problem?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061035838


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);
+    }
+    System.arraycopy(text, 0, writeBuffer, writePos, text.length);
+    writePos += text.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.NULL);
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(b ? VariantUtil.TRUE : VariantUtil.FALSE);
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder. The actual encoded integer type depends on the
+   * value range of the long value.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    if (l == (byte) l) {
+      checkCapacity(1 + 1);

Review Comment:
   Without an inline comment for context, this is a bit odd. I think it would make sense to use either `1 /* header size */ + 1` or just switch to constants. That is, without a comment, I don't think this is more clear than `checkCapacity(2)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061012912


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;

Review Comment:
   The variable name here doesn't quite make sense to me because this isn't the size of a string, it is the size of all strings in the data/content section of the dictionary. What about `totalDataSize` or `totalContentSize`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061012912


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;

Review Comment:
   The variable name here doesn't quite make sense to me. What about `totalDataSize` or `totalContentSize`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2070750210


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);

Review Comment:
   When setting instance fields, it helps to add the prefix `this.` so the reader can easily see it isn't a local variable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue merged PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069319424


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantArrayBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant arrays, used by VariantBuilder.
+ */
+public class VariantArrayBuilder {
+  VariantArrayBuilder(VariantBuilder builder) {
+    this.builder = builder;
+    this.startPos = builder.writePos();
+    this.offsets = new ArrayList<>();
+  }
+
+  void startElement() {
+    offsets.add(builder.writePos() - startPos);
+  }
+
+  int startPos() {
+    return startPos;
+  }
+
+  ArrayList<Integer> offsets() {
+    return offsets;
+  }
+
+  /** The underlying VariantBuilder. */

Review Comment:
   It's more common to list instance fields at the top of a class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069423391


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -187,6 +187,29 @@ class VariantUtil {
   // The size (in bytes) of a UUID.
   static final int UUID_SIZE = 16;
 
+  // header bytes
+  static final byte HDR_NULL = primitiveHeader(NULL);

Review Comment:
   Nit: Omitting vowels doesn't help much. The code is less readable and it's not actually easier to type hdr than it is to type header.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["gene-db (via GitHub)" <gi...@apache.org>]

gene-db commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2066974514


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);
+    }
+    System.arraycopy(text, 0, writeBuffer, writePos, text.length);
+    writePos += text.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.NULL);
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(b ? VariantUtil.TRUE : VariantUtil.FALSE);
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder. The actual encoded integer type depends on the
+   * value range of the long value.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {

Review Comment:
   Updated this and introduced `appendInt`, `appendShort`, `appendByte`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073893937


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,631 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    // TODO: Reduce the copying, and look into builder reuse.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    this.objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset, because it was written last.
+        if (fields.get(distinctPos).offset < fields.get(i).offset) {
+          fields.set(distinctPos, fields.get(i));
+        }
+      } else {
+        // Found a distinct key. Add the field to the list.
+        distinctPos++;
+        fields.set(distinctPos, fields.get(i));
+        dataSize += fields.get(i).valueSize;
+      }
+    }
+
+    if (distinctPos + 1 < fields.size()) {
+      numFields = distinctPos + 1;
+      // Resize `fields` to `size`.
+      fields.subList(numFields, fields.size()).clear();
+    }
+
+    boolean largeSize = numFields > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The data starts after: the header byte, object size, id list, and offset list.
+    int dataOffset = 1 + sizeBytes + numFields * idSize + (numFields + 1) * offsetSize;
+    checkCapacity(dataOffset + dataSize);
+
+    // Write the header byte and size entry.
+    writeBuffer[writePos] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numFields, sizeBytes);
+
+    int idStart = writePos + 1 + sizeBytes;
+    int offsetStart = idStart + numFields * idSize;
+    int currOffset = 0;
+
+    // Loop over all fields and write the key id, offset, and data to the appropriate offsets
+    for (int i = 0; i < numFields; ++i) {
+      FieldEntry field = fields.get(i);
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, field.id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, currOffset, offsetSize);
+      System.arraycopy(
+          objectBuilder.writeBuffer,
+          field.offset,
+          writeBuffer,
+          writePos + dataOffset + currOffset,
+          field.valueSize);
+      currOffset += field.valueSize;
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numFields * offsetSize, dataSize, offsetSize);
+    writePos += dataOffset + dataSize;
+    this.objectBuilder = null;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder is used to
+   * append values ot the array. startArray() must be called before endArray(). No append*() methods
+   * can be called in between startArray() and endArray().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.appendString("value1");
+   * arrayBuilder.appendString("value2");
+   * builder.endArray();
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endArray() first.");
+    }
+    this.arrayBuilder = new VariantArrayBuilder(this);
+    return arrayBuilder;
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   */
+  public void endArray() {
+    if (arrayBuilder == null) {
+      throw new IllegalStateException("Cannot call endArray() without calling startArray() first.");
+    }
+    ArrayList<Integer> offsets = arrayBuilder.validateAndGetOffsets();
+    int numElements = offsets.size();
+    int dataSize = arrayBuilder.writePos;
+    boolean largeSize = numElements > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The data starts after: the header byte, object size, and offset list.
+    int dataOffset = 1 + sizeBytes + (numElements + 1) * offsetSize;
+    checkCapacity(dataOffset + dataSize);
+
+    // Copy all the element data to the write buffer.
+    System.arraycopy(arrayBuilder.writeBuffer, 0, writeBuffer, writePos + dataOffset, dataSize);
+
+    writeBuffer[writePos] = VariantUtil.arrayHeader(largeSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numElements, sizeBytes);
+
+    int offsetStart = writePos + 1 + sizeBytes;
+    for (int i = 0; i < numElements; ++i) {
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, offsets.get(i), offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numElements * offsetSize, dataSize, offsetSize);
+    writePos += dataOffset + dataSize;
+    this.arrayBuilder = null;
+  }
+
+  protected void onAppend() {
+    checkAppendWhileNested();
+    if (writePos > 0) {
+      throw new IllegalStateException("Cannot call multiple append() methods.");
+    }
+  }
+
+  protected void onStartNested() {
+    checkMultipleNested("Cannot call startObject()/startArray() without calling endObject()/endArray() first.");
+  }
+
+  protected void checkMultipleNested(String message) {
+    if (objectBuilder != null || arrayBuilder != null) {
+      throw new IllegalStateException(message);
+    }
+  }
+
+  protected void checkAppendWhileNested() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call append() methods while an object is being built. Must call endObject() first.");

Review Comment:
   I don't think this exception message is very helpful because the API is a bit confusing. Some similar builders (like Jackson) would change the state of the current builder, rather than forcing the caller to use a new builder that is returned. It's okay to make this choice about how the builders are related, but the error message needs to tell the caller what to do.
   
   In this case, the caller should be informed that `append` methods must be called on the builder returned by `startArray` and `endArray` is called on the original builder when those are complete. If you don't know that you should be using a different builder, then "cannot call append methods while an object is being built" doesn't make sense. "Append object fields using appendKey and append methods on the builder returned by startObject and call endObject on this builder" Or something like that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069545094


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -137,136 +118,248 @@ public void testNegativeLong() {
           (byte) 0xFF,
           (byte) 0xFF
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
       Assert.assertEquals(-1L, v.getLong());
     });
   }
 
   @Test
   public void testInt() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(1234567890, v.getInt());
     });
   }
 
   @Test
   public void testNegativeInt() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(5), (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF}),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+        ByteBuffer.wrap(new byte[] {
+          VariantTestUtil.primitiveHeader(5), (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF
+        }),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(-1, v.getInt());
     });
   }
 
   @Test
   public void testShort() {
-    Variant value =
-        new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(4), (byte) 0xD2, 0x04}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(4), (byte) 0xD2, 0x04}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
       Assert.assertEquals((short) 1234, v.getShort());
     });
   }
 
   @Test
   public void testNegativeShort() {
-    Variant value =
-        new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(4), (byte) 0xFF, (byte) 0xFF}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(4), (byte) 0xFF, (byte) 0xFF}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
       Assert.assertEquals((short) -1, v.getShort());
     });
   }
 
   @Test
   public void testByte() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(3), 34}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(3), 34}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
       Assert.assertEquals((byte) 34, v.getByte());
     });
   }
 
   @Test
   public void testNegativeByte() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(3), (byte) 0xFF}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(3), (byte) 0xFF}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
       Assert.assertEquals((byte) -1, v.getByte());
     });
   }
 
+  @Test
+  public void testLongBuilder() {
+    Arrays.asList(
+            0L,
+            (long) Byte.MIN_VALUE,
+            (long) Byte.MAX_VALUE,
+            (long) Short.MIN_VALUE,
+            (long) Short.MAX_VALUE,
+            (long) Integer.MIN_VALUE,
+            (long) Integer.MAX_VALUE,
+            Long.MIN_VALUE,
+            Long.MAX_VALUE)
+        .forEach(l -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendLong(l);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
+            Assert.assertEquals((long) l, v.getLong());
+          });
+        });
+  }
+
+  @Test
+  public void testIntBuilder() {
+    Arrays.asList(
+            0,
+            (int) Byte.MIN_VALUE,
+            (int) Byte.MAX_VALUE,
+            (int) Short.MIN_VALUE,
+            (int) Short.MAX_VALUE,
+            Integer.MIN_VALUE,
+            Integer.MAX_VALUE)
+        .forEach(i -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendInt(i);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+            Assert.assertEquals((int) i, v.getInt());
+          });
+        });
+  }
+
+  @Test
+  public void testShortBuilder() {
+    Arrays.asList((short) 0, (short) Byte.MIN_VALUE, (short) Byte.MAX_VALUE, Short.MIN_VALUE, Short.MAX_VALUE)
+        .forEach(s -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendShort(s);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+            Assert.assertEquals((short) s, v.getShort());
+          });
+        });
+  }
+
+  @Test
+  public void testByteBuilder() {
+    Arrays.asList((byte) 0, Byte.MIN_VALUE, Byte.MAX_VALUE).forEach(b -> {
+      VariantBuilder vb2 = new VariantBuilder(false);
+      vb2.appendByte(b);
+      VariantTestUtil.testVariant(vb2.build(), v -> {
+        VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+        Assert.assertEquals((byte) b, v.getByte());
+      });
+    });
+  }
+
   @Test
   public void testFloat() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(14), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(14), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
       Assert.assertEquals(Float.intBitsToFloat(1234567890), v.getFloat(), 0);
     });
   }
 
   @Test
   public void testNegativeFloat() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(14), 0x00, 0x00, 0x00, (byte) 0x80}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(14), 0x00, 0x00, 0x00, (byte) 0x80}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
       Assert.assertEquals(-0.0F, v.getFloat(), 0);
     });
   }
 
+  @Test
+  public void testFloatBuilder() {
+    Arrays.asList(Float.MIN_VALUE, 0f, Float.MAX_VALUE).forEach(f -> {

Review Comment:
   -0f?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069554859


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantObjectBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant object, used by VariantBuilder.
+ */
+public class VariantObjectBuilder {

Review Comment:
   I think that this builder should be used for both keys and values so that it can help verify that there are the same number of keys and values. Right now, there is no checking and it isn't clear when values should be added. Meaning the API doesn't help show me that `appendKey` should be interleaved with `appendSomeValue` calls. If I leave out the `appendSomeValue` call then things break. I think there should be checks in this API that ensure correct object construction. There should probably also be tests for the weird cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2070745623


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {

Review Comment:
   This is a note to myself that I think this is implemented correctly so I don't need to review it again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069421223


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {
+      int distinctPos = 0;
+      // Maintain a list of distinct keys in-place.
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        if (fields.get(i).id == fields.get(i - 1).id) {
+          // Found a duplicate key. Keep the field with the greater offset.
+          if (fields.get(distinctPos).offset < fields.get(i).offset) {
+            fields.set(distinctPos, fields.get(distinctPos).withNewOffset(fields.get(i).offset));
+          }
+        } else {
+          // Found a distinct key. Add the field to the list.
+          ++distinctPos;
+          fields.set(distinctPos, fields.get(i));
+        }
+      }
+      if (distinctPos + 1 < fields.size()) {
+        size = distinctPos + 1;
+        // Resize `fields` to `size`.
+        fields.subList(size, fields.size()).clear();
+        // Sort the fields by offsets so that we can move the value data of each field to the new
+        // offset without overwriting the fields after it.
+        fields.sort(Comparator.comparingInt(f -> f.offset));
+        int currentOffset = 0;
+        for (int i = 0; i < size; ++i) {
+          int oldOffset = fields.get(i).offset;
+          int fieldSize =
+              VariantUtil.valueSize(ByteBuffer.wrap(writeBuffer), objBuilder.startPos() + oldOffset);
+          System.arraycopy(
+              writeBuffer,
+              objBuilder.startPos() + oldOffset,
+              writeBuffer,
+              objBuilder.startPos() + currentOffset,
+              fieldSize);
+          fields.set(i, fields.get(i).withNewOffset(currentOffset));
+          currentOffset += fieldSize;
+        }
+        writePos = objBuilder.startPos() + currentOffset;
+        // Change back to the sort order by field keys, required by the Variant specification.
+        Collections.sort(fields);
+      }
+    } else {
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        String key = fields.get(i).key;
+        if (key.equals(fields.get(i - 1).key)) {
+          throw new IllegalStateException("Failed to build Variant because of duplicate object key: " + key);
+        }
+      }
+    }
+    int dataSize = writePos - objBuilder.startPos();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + size * idSize + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the object header section.
+    System.arraycopy(writeBuffer, objBuilder.startPos(), writeBuffer, objBuilder.startPos() + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[objBuilder.startPos()] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, objBuilder.startPos() + 1, size, sizeBytes);
+    int idStart = objBuilder.startPos() + 1 + sizeBytes;
+    int offsetStart = idStart + size * idSize;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, fields.get(i).id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, fields.get(i).offset, offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder must be
+   * used for future calls to startArrayElement() and endArray(). To add an element to the array,
+   * call startArrayElement() and then append the value. endArray() must be called to finish writing
+   * the Variant array.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.startArrayElement(arrayBuilder);
+   * builder.appendString("value1");
+   * arrayBuilder.endArray(arrayBuilder);
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    return new VariantArrayBuilder(this);
+  }
+
+  /**
+   * Starts appending an element to the array. This method must be called before appending the
+   * corresponding value.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   */
+  public void startArrayElement(VariantArrayBuilder arrayBuilder) {
+    arrayBuilder.startElement();
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endArray(VariantArrayBuilder arrayBuilder) {
+    int start = arrayBuilder.startPos();
+    int dataSize = writePos - start;
+    int size = arrayBuilder.offsets().size();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the header section.
+    System.arraycopy(writeBuffer, start, writeBuffer, start + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[start] = VariantUtil.arrayHeader(largeSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, start + 1, size, sizeBytes);
+    int offsetStart = start + 1 + sizeBytes;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(
+          writeBuffer,
+          offsetStart + i * offsetSize,
+          arrayBuilder.offsets().get(i),
+          offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Adds a key to the Variant dictionary. If the key already exists, the dictionary is unmodified.
+   * @param key the key to add
+   * @return the id of the key
+   */
+  int addKey(String key) {
+    return dictionary.computeIfAbsent(key, newKey -> {
+      int id = dictionaryKeys.size();
+      dictionaryKeys.add(newKey.getBytes(StandardCharsets.UTF_8));
+      return id;
+    });
+  }
+
+  /**
+   * @return the current write position of the variant builder
+   */
+  int writePos() {
+    return writePos;
+  }
+
+  /**
+   * Class to store the information of a Variant object field. We need to collect all fields of
+   * an object, sort them by their keys, and build the Variant object in sorted order.
+   */
+  static final class FieldEntry implements Comparable<FieldEntry> {
+    final String key;
+    final int id;
+    final int offset;
+
+    FieldEntry(String key, int id, int offset) {
+      this.key = key;
+      this.id = id;
+      this.offset = offset;
+    }
+
+    FieldEntry withNewOffset(int newOffset) {
+      return new FieldEntry(key, id, newOffset);
+    }
+
+    @Override
+    public int compareTo(FieldEntry other) {
+      return key.compareTo(other.key);
+    }
+  }
+
+  private void checkCapacity(int additionalBytes) {
+    int requiredBytes = writePos + additionalBytes;
+    if (requiredBytes > writeBuffer.length) {
+      // Allocate a new buffer with a capacity of the next power of 2 of `requiredBytes`.
+      int newCapacity = Integer.highestOneBit(requiredBytes);
+      newCapacity = newCapacity < requiredBytes ? newCapacity * 2 : newCapacity;
+      byte[] newValue = new byte[newCapacity];
+      System.arraycopy(writeBuffer, 0, newValue, 0, writePos);
+      writeBuffer = newValue;
+    }
+  }
+
+  private int getMinIntegerSize(int value) {
+    assert value >= 0;
+    if (value <= VariantUtil.U8_MAX) {
+      return VariantUtil.U8_SIZE;
+    }
+    if (value <= VariantUtil.U16_MAX) {
+      return VariantUtil.U16_SIZE;
+    }
+    if (value <= VariantUtil.U24_MAX) {
+      return VariantUtil.U24_SIZE;
+    }
+    return VariantUtil.U32_SIZE;
+  }
+
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  private byte[] writeBuffer = new byte[128];

Review Comment:
   If this is trying to avoid allocations, then I think this should start at a larger size. It would make sense to me to start a 1k or 2k, considering this is probably reused and we don't want to reallocate very often.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2072161635


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);

Review Comment:
   Right now, if the builder isn't going to be reused then there's no point in copying. It would be better to use a buffer since we already know the length of the variant value in the array (`writePos`).
   
   If you wanted to reuse the builder then I think this would change a little. We want a reused builder to keep track of its underlying buffer so that it doesn't need to grow each time it creates a new value (assuming that we have somewhat uniform value sizes). In that case, I think you'd want to copy the value out of the reused buffer that is owned by the builder.
   
   Builder reuse is something we can work on later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["gene-db (via GitHub)" <gi...@apache.org>]

gene-db commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2064070328


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.

Review Comment:
   Renamed.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {

Review Comment:
   It looks incorrect or correct? If incorrect, which part is incorrect?



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);

Review Comment:
   Renamed to `data`.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);

Review Comment:
   That makes sense. I updated this and others to avoid this pattern.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;

Review Comment:
   Yeah, makes sense. Renamed.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);

Review Comment:
   This initial implementation, yes. I left a TODO comment to add key sorting.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);
+    }
+    System.arraycopy(text, 0, writeBuffer, writePos, text.length);
+    writePos += text.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.NULL);
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(b ? VariantUtil.TRUE : VariantUtil.FALSE);
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder. The actual encoded integer type depends on the
+   * value range of the long value.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    if (l == (byte) l) {
+      checkCapacity(1 + 1);
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.INT8);
+      VariantUtil.writeLong(writeBuffer, writePos, l, 1);
+      writePos += 1;
+    } else if (l == (short) l) {
+      checkCapacity(1 + 2);
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.INT16);
+      VariantUtil.writeLong(writeBuffer, writePos, l, 2);

Review Comment:
   Updated.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);

Review Comment:
   The original reason to copy out was to only keep the required length and not the full capacity.
   
   What do you think? Should we just keep the buffer in the builder, and point to it with a `ByteBuffer`?



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;

Review Comment:
   Renamed.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);
+    }
+    System.arraycopy(text, 0, writeBuffer, writePos, text.length);
+    writePos += text.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.NULL);
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(b ? VariantUtil.TRUE : VariantUtil.FALSE);
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder. The actual encoded integer type depends on the
+   * value range of the long value.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    if (l == (byte) l) {
+      checkCapacity(1 + 1);

Review Comment:
   Added a comment to this and other similar instances.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);

Review Comment:
   Created constants for all these headers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061031114


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);

Review Comment:
   Why copy here? If the buffer may be reused, then a comment would help clarify the contract.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061027202


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);

Review Comment:
   Always unsorted?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073868996


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantArrayBuilder.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestVariantArrayBuilder {
+  private static final Logger LOG = LoggerFactory.getLogger(TestVariantArrayBuilder.class);
+
+  @Test
+  public void testEmptyArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, v.numArrayElements());
+    });
+  }
+
+  @Test
+  public void testLargeArraySizeBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    for (int i = 0; i < 511; i++) {
+      a.appendInt(i);
+    }
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(511, v.numArrayElements());
+      for (int i = 0; i < 511; i++) {
+        VariantTestUtil.checkType(v.getElementAtIndex(i), VariantUtil.PRIMITIVE, Variant.Type.INT);
+        Assert.assertEquals(i, v.getElementAtIndex(i).getInt());
+      }
+    });
+  }
+
+  @Test
+  public void testMixedArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendBoolean(true);
+    VariantObjectBuilder obj = arrBuilder.startObject();
+    obj.appendKey("key");
+    obj.appendInt(321);
+    arrBuilder.endObject();
+    arrBuilder.appendLong(1234567890);
+    {
+      // build a nested array
+      VariantArrayBuilder nestedBuilder = arrBuilder.startArray();
+      {
+        // build a nested empty array
+        nestedBuilder.startArray();
+        nestedBuilder.endArray();
+      }
+      nestedBuilder.appendString("variant");
+      nestedBuilder.startObject();
+      nestedBuilder.endObject();
+      arrBuilder.endArray();
+    }
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(4, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(0).getBoolean());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(1, v.getElementAtIndex(1).numObjectElements());
+      VariantTestUtil.checkType(
+          v.getElementAtIndex(1).getFieldByKey("key"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      Assert.assertEquals(321, v.getElementAtIndex(1).getFieldByKey("key").getInt());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(3), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Variant nested = v.getElementAtIndex(3);
+      Assert.assertEquals(3, nested.numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(0), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, nested.getElementAtIndex(0).numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(1), VariantUtil.SHORT_STR, Variant.Type.STRING);
+      Assert.assertEquals("variant", nested.getElementAtIndex(1).getString());
+      VariantTestUtil.checkType(nested.getElementAtIndex(2), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, nested.getElementAtIndex(2).numObjectElements());
+    });
+  }
+
+  private void buildNested(int i, VariantArrayBuilder obj) {
+    if (i > 0) {
+      obj.appendString("str" + i);
+      buildNested(i - 1, obj.startArray());
+      obj.endArray();
+    }
+  }
+
+  @Test
+  public void testNestedBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    buildNested(1000, b.startArray());
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      Variant curr = v;
+      for (int i = 1000; i >= 0; i--) {
+        VariantTestUtil.checkType(curr, VariantUtil.ARRAY, Variant.Type.ARRAY);
+        if (i == 0) {
+          Assert.assertEquals(0, curr.numArrayElements());
+        } else {
+          Assert.assertEquals(2, curr.numArrayElements());
+          VariantTestUtil.checkType(curr.getElementAtIndex(0), VariantUtil.SHORT_STR, Variant.Type.STRING);
+          Assert.assertEquals("str" + i, curr.getElementAtIndex(0).getString());
+          curr = curr.getElementAtIndex(1);
+        }
+      }
+    });
+  }
+
+  private void testArrayOffsetSizeBuilder(String randomString) {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendString(randomString);
+    arrBuilder.appendBoolean(true);
+    arrBuilder.appendLong(1234567890);
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(3, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.STRING);
+      Assert.assertEquals(randomString, v.getElementAtIndex(0).getString());
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(1).getBoolean());
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+    });
+  }
+
+  @Test
+  public void testArrayTwoByteOffsetBuilder() {
+    // a string larger than 255 bytes to push the value offset size above 1 byte
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(300));
+  }
+
+  @Test
+  public void testArrayThreeByteOffsetBuilder() {
+    // a string larger than 65535 bytes to push the value offset size above 2 bytes
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(70_000));
+  }
+
+  @Test
+  public void testArrayFourByteOffsetBuilder() {
+    // a string larger than 16777215 bytes to push the value offset size above 3 bytes
+    testArrayOffsetSizeBuilder(VariantTestUtil.randomString(16_800_000));
+  }
+
+  @Test
+  public void testMissingEndArray() {
+    VariantBuilder b = new VariantBuilder();
+    b.startArray();
+    try {
+      b.build();
+      Assert.fail("Expected Exception when calling build() without endArray()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testMissingStartArray() {
+    VariantBuilder b = new VariantBuilder();
+    try {
+      b.endArray();
+      Assert.fail("Expected Exception when calling endArray() without startArray()");
+    } catch (Exception e) {
+      // expected
+    }
+  }
+
+  @Test
+  public void testInvalidAppendDuringArray() {
+    VariantBuilder b = new VariantBuilder();
+    b.startArray();
+    try {
+      b.appendInt(1);
+      Assert.fail("Expected Exception when calling append() before endArray()");

Review Comment:
   I don't think this exception message is helpful because the API is a bit confusing. Some similar builders (like Jackson) would change the state of the current builder, rather than forcing the caller to use a new builder that is returned. It's okay to make this choice about how the builders are related, but the error message needs to tell the caller what to do. In this case, the caller should be informed that `append` methods must be called on the builder returned by `startArray` and `endArray` is called on the original builder when those are complete.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069548151


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -506,32 +730,54 @@ public void testTime() {
           (byte) 0x00,
           (byte) 0x00
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIME);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIME);
       Assert.assertEquals(LocalTime.parse("23:59:59.123456"), LocalTime.ofNanoOfDay(v.getLong() * 1_000));
     });
   }
 
   @Test
-  public void testTimestampNanos() {
+  public void testTimeBuilder() {
+    for (String timeStr : Arrays.asList(
+        "00:00:00.000000", "00:00:00.000120", "12:00:00.000000", "12:00:00.002300", "23:59:59.999999")) {
+      VariantBuilder vb = new VariantBuilder(false);
+      long micros = LocalTime.parse(timeStr).toNanoOfDay() / 1_000;
+      vb.appendTime(micros);

Review Comment:
   Should this allow or reject negative values?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["aihuaxu (via GitHub)" <gi...@apache.org>]

aihuaxu commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069805783


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -34,91 +37,69 @@
 public class TestVariantScalar {
   private static final Logger LOG = LoggerFactory.getLogger(TestVariantScalar.class);
 
-  private static final ByteBuffer EMPTY_METADATA = ByteBuffer.wrap(new byte[] {0b1});
-
-  private void checkType(Variant v, int expectedBasicType, Variant.Type expectedType) {
-    Assert.assertEquals(expectedBasicType, v.value.get(v.value.position()) & VariantUtil.BASIC_TYPE_MASK);
-    Assert.assertEquals(expectedType, v.getType());
-  }
-
-  private void testVariant(Variant v, Consumer<Variant> consumer) {
-    consumer.accept(v);
-    // Create new Variant with different byte offsets
-    byte[] newValue = new byte[v.value.capacity() + 50];
-    byte[] newMetadata = new byte[v.metadata.capacity() + 50];
-    Arrays.fill(newValue, (byte) 0xFF);
-    Arrays.fill(newMetadata, (byte) 0xFF);
-    v.value.position(0);
-    v.value.get(newValue, 25, v.value.capacity());
-    v.value.position(0);
-    v.metadata.position(0);
-    v.metadata.get(newMetadata, 25, v.metadata.capacity());
-    v.metadata.position(0);
-    Variant v2 = new Variant(
-        ByteBuffer.wrap(newValue, 25, v.value.capacity()),
-        ByteBuffer.wrap(newMetadata, 25, v.metadata.capacity()));
-    consumer.accept(v2);
-  }
-
-  private static byte primitiveHeader(int type) {
-    return (byte) (type << 2);
-  }
-
-  private static byte metadataHeader(boolean isSorted, int offsetSize) {
-    return (byte) (((offsetSize - 1) << 6) | (isSorted ? 0b10000 : 0) | 0b0001);
-  }
-
-  private static int getMinIntegerSize(int value) {
-    return (value <= 0xFF) ? 1 : (value <= 0xFFFF) ? 2 : (value <= 0xFFFFFF) ? 3 : 4;
-  }
-
-  private static void writeVarlenInt(ByteBuffer buffer, int value, int valueSize) {
-    if (valueSize == 1) {
-      buffer.put((byte) value);
-    } else if (valueSize == 2) {
-      buffer.putShort((short) value);
-    } else if (valueSize == 3) {
-      buffer.put((byte) (value & 0xFF));
-      buffer.put((byte) ((value >> 8) & 0xFF));
-      buffer.put((byte) ((value >> 16) & 0xFF));
-    } else {
-      buffer.putInt(value);
-    }
+  @Test
+  public void testNull() {
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(0)}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> VariantTestUtil.checkType(v, VariantUtil.NULL, Variant.Type.NULL));
   }
 
   @Test
-  public void testNull() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(0)}), EMPTY_METADATA);
-    testVariant(value, v -> checkType(v, VariantUtil.NULL, Variant.Type.NULL));
+  public void testNullBuilder() {
+    VariantBuilder vb = new VariantBuilder(false);
+    vb.appendNull();

Review Comment:
   Here we append null and then we can still append other values.
   Maybe we can have VariantBuilder.ofNull() or similar to prevent this? 
   
   



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObject.java:
##########
@@ -205,22 +137,52 @@ private static ByteBuffer constructMetadata(Boolean isSorted, List<String> field
 
   @Test
   public void testEmptyObject() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {0b10, 0x00}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    Variant value = new Variant(ByteBuffer.wrap(new byte[] {0b10, 0x00}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, v.numObjectElements());
+    });
+  }
+
+  @Test
+  public void testEmptyObjectBuilder() {
+    VariantBuilder b = new VariantBuilder(true);
+    VariantObjectBuilder o = b.startObject();
+    b.endObject(o);
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(0, v.numObjectElements());
     });
   }
 
   @Test
   public void testEmptyLargeObject() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {0b1000010, 0x00, 0x00, 0x00, 0x00}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {0b1000010, 0x00, 0x00, 0x00, 0x00}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(0, v.numObjectElements());
     });
   }
 
+  @Test
+  public void testLargeObjectBuilder() {
+    VariantBuilder b = new VariantBuilder(true);
+    VariantObjectBuilder o = b.startObject();
+    for (int i = 0; i < 1234; i++) {
+      b.appendObjectKey(o, "a" + i);
+      b.appendLong(i);
+    }
+    b.endObject(o);

Review Comment:
   I also did the similar implementation before to have endObject() and endArray(), but the caller needs to be careful to call in certain sequence.
   
   We probably can have something similar tohttps://docs.oracle.com/javaee/7/api/javax/json/JsonObjectBuilder.html and JsonArrayBuilder.
   
   
   
   
   



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantArrayBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant arrays, used by VariantBuilder.

Review Comment:
   @gene-db  Seems we didn't include the parseJson() implementation which your original PR did. I guess we will implement separately, is that right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2070934255


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset.
+        if (fields.get(distinctPos).offset < fields.get(i).offset) {
+          fields.set(distinctPos, fields.get(i));
+        }
+      } else {
+        // Found a distinct key. Add the field to the list.
+        distinctPos++;
+        fields.set(distinctPos, fields.get(i));
+        dataSize += fields.get(i).valueSize;
+      }
+    }
+
+    if (distinctPos + 1 < fields.size()) {
+      numFields = distinctPos + 1;
+      // Resize `fields` to `size`.
+      fields.subList(numFields, fields.size()).clear();
+    }
+
+    boolean largeSize = numFields > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + numFields * idSize + (numFields + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Write the header byte and size entry.
+    writeBuffer[writePos] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numFields, sizeBytes);
+
+    int idStart = writePos + 1 + sizeBytes;
+    int offsetStart = idStart + numFields * idSize;
+    int fieldDataStart = offsetStart + (numFields + 1) * offsetSize;
+    int currOffset = 0;
+
+    // Loop over all fields and write the key id, offset, and data to the appropriate offsets
+    for (int i = 0; i < numFields; ++i) {
+      FieldEntry field = fields.get(i);
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, field.id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, currOffset, offsetSize);
+      System.arraycopy(
+          objectBuilder.writeBuffer, field.offset, writeBuffer, fieldDataStart + currOffset, field.valueSize);
+      currOffset += field.valueSize;
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numFields * offsetSize, dataSize, offsetSize);
+    writePos += headerSize + dataSize;
+    numValues++;
+    objectBuilder = null;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder is used to
+   * append values ot the array. startArray() must be called before endArray(). No append*() methods
+   * can be called in between startArray() and endArray().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.appendString("value1");
+   * arrayBuilder.appendString("value2");
+   * builder.endArray();
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endArray() first.");
+    }
+    arrayBuilder = new VariantArrayBuilder(this);
+    return arrayBuilder;
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   */
+  public void endArray() {
+    if (arrayBuilder == null) {
+      throw new IllegalStateException("Cannot call endArray() without calling startArray() first.");
+    }
+    ArrayList<Integer> offsets = arrayBuilder.validateAndGetOffsets();
+    int numElements = offsets.size();
+    int dataSize = arrayBuilder.writePos;
+    boolean largeSize = numElements > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (numElements + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Copy all the element data to the write buffer.
+    System.arraycopy(arrayBuilder.writeBuffer, 0, writeBuffer, writePos + headerSize, dataSize);

Review Comment:
   Yeah, this calls `arrayHeader` in the next statement, which produces a single byte.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2072166725


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,658 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    incrementNumValues();

Review Comment:
   After looking into `endObject` and `endArray`, I think the rationale for incrementing values at the end of each append is to make the consistency check in `endObject` catch cases where a nested object is partially written. So if you have a nested object, the parent won't have a matching number of keys and values until the inner `endObject` is called.
   
   I think we need a stronger validation that doesn't rely on when `incrementNumValues` is called. First, I don't think that check would similarly work when an object in an array is unfinished. So the `validate` methods for both arrays and objects need to call `checkMultipleNested` to verify that there are no unfinished objects or arrays in the child builder.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["gene-db (via GitHub)" <gi...@apache.org>]

gene-db commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2072179064


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantArrayBuilder.java:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant arrays, used by VariantBuilder.
+ */
+public class VariantArrayBuilder extends VariantBuilder {
+  /** The parent VariantBuilder. */
+  private final VariantBuilder parent;
+  /** The offsets of the elements in this array. */
+  private final ArrayList<Integer> offsets;
+  /** The number of values appended to this array. */
+  protected long numValues = 0;
+
+  VariantArrayBuilder(VariantBuilder parent) {
+    this.parent = parent;
+    this.offsets = new ArrayList<>();
+  }
+
+  /**
+   * @return the list of element offsets in this array
+   */
+  ArrayList<Integer> validateAndGetOffsets() {
+    if (offsets.size() != numValues) {
+      throw new IllegalStateException(String.format(
+          "Number of offsets (%d) do not match the number of values (%d).", offsets.size(), numValues));
+    }
+    return offsets;
+  }
+
+  @Override
+  protected void onAppend() {
+    checkAppendWhileNested();
+    offsets.add(writePos);
+  }
+
+  @Override
+  protected void onStartNested() {
+    checkMultipleNested();
+    offsets.add(writePos);
+  }
+
+  @Override
+  protected void incrementNumValues() {

Review Comment:
   Moved to `onAppend`.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantObjectBuilder.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant object, used by VariantBuilder.
+ */
+public class VariantObjectBuilder extends VariantBuilder {
+  /** The parent VariantBuilder. */
+  private final VariantBuilder parent;
+  /** The FieldEntry list for the fields of this object. */
+  private final ArrayList<VariantBuilder.FieldEntry> fields;
+  /** The number of values appended to this object. */
+  protected long numValues = 0;
+
+  VariantObjectBuilder(VariantBuilder parent) {
+    this.parent = parent;
+    this.fields = new ArrayList<>();
+  }
+
+  /**
+   * Appends an object key to this object. This method must be called before appending any value.
+   * @param key the key to append
+   */
+  void appendKey(String key) {
+    if (fields.size() > numValues) {
+      throw new IllegalStateException("Cannot call appendKey() before appending an object value.");

Review Comment:
   Yes, updated.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);

Review Comment:
   Ok. Added a TODO to mention this.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,658 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    incrementNumValues();

Review Comment:
   Added that additional validation for both the array builder and object builder, and added tests for those scenarios.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073877840


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantArrayBuilder.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestVariantArrayBuilder {
+  private static final Logger LOG = LoggerFactory.getLogger(TestVariantArrayBuilder.class);
+
+  @Test
+  public void testEmptyArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, v.numArrayElements());
+    });
+  }
+
+  @Test
+  public void testLargeArraySizeBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    for (int i = 0; i < 511; i++) {
+      a.appendInt(i);
+    }
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(511, v.numArrayElements());
+      for (int i = 0; i < 511; i++) {
+        VariantTestUtil.checkType(v.getElementAtIndex(i), VariantUtil.PRIMITIVE, Variant.Type.INT);
+        Assert.assertEquals(i, v.getElementAtIndex(i).getInt());
+      }
+    });
+  }
+
+  @Test
+  public void testMixedArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendBoolean(true);
+    VariantObjectBuilder obj = arrBuilder.startObject();
+    obj.appendKey("key");
+    obj.appendInt(321);
+    arrBuilder.endObject();
+    arrBuilder.appendLong(1234567890);
+    {
+      // build a nested array
+      VariantArrayBuilder nestedBuilder = arrBuilder.startArray();
+      {
+        // build a nested empty array
+        nestedBuilder.startArray();
+        nestedBuilder.endArray();
+      }
+      nestedBuilder.appendString("variant");
+      nestedBuilder.startObject();
+      nestedBuilder.endObject();
+      arrBuilder.endArray();
+    }
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(4, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(0).getBoolean());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(1, v.getElementAtIndex(1).numObjectElements());
+      VariantTestUtil.checkType(
+          v.getElementAtIndex(1).getFieldByKey("key"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      Assert.assertEquals(321, v.getElementAtIndex(1).getFieldByKey("key").getInt());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(3), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Variant nested = v.getElementAtIndex(3);
+      Assert.assertEquals(3, nested.numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(0), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, nested.getElementAtIndex(0).numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(1), VariantUtil.SHORT_STR, Variant.Type.STRING);
+      Assert.assertEquals("variant", nested.getElementAtIndex(1).getString());
+      VariantTestUtil.checkType(nested.getElementAtIndex(2), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, nested.getElementAtIndex(2).numObjectElements());
+    });
+  }
+
+  private void buildNested(int i, VariantArrayBuilder obj) {

Review Comment:
   I find this way too hard to reason about for a test. Can you rewrite it to a for loop in `testNestedBuilder`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2073877840


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantArrayBuilder.java:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class TestVariantArrayBuilder {
+  private static final Logger LOG = LoggerFactory.getLogger(TestVariantArrayBuilder.class);
+
+  @Test
+  public void testEmptyArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, v.numArrayElements());
+    });
+  }
+
+  @Test
+  public void testLargeArraySizeBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder a = b.startArray();
+    for (int i = 0; i < 511; i++) {
+      a.appendInt(i);
+    }
+    b.endArray();
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(511, v.numArrayElements());
+      for (int i = 0; i < 511; i++) {
+        VariantTestUtil.checkType(v.getElementAtIndex(i), VariantUtil.PRIMITIVE, Variant.Type.INT);
+        Assert.assertEquals(i, v.getElementAtIndex(i).getInt());
+      }
+    });
+  }
+
+  @Test
+  public void testMixedArrayBuilder() {
+    VariantBuilder b = new VariantBuilder();
+    VariantArrayBuilder arrBuilder = b.startArray();
+    arrBuilder.appendBoolean(true);
+    VariantObjectBuilder obj = arrBuilder.startObject();
+    obj.appendKey("key");
+    obj.appendInt(321);
+    arrBuilder.endObject();
+    arrBuilder.appendLong(1234567890);
+    {
+      // build a nested array
+      VariantArrayBuilder nestedBuilder = arrBuilder.startArray();
+      {
+        // build a nested empty array
+        nestedBuilder.startArray();
+        nestedBuilder.endArray();
+      }
+      nestedBuilder.appendString("variant");
+      nestedBuilder.startObject();
+      nestedBuilder.endObject();
+      arrBuilder.endArray();
+    }
+    b.endArray();
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(4, v.numArrayElements());
+      VariantTestUtil.checkType(v.getElementAtIndex(0), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      Assert.assertTrue(v.getElementAtIndex(0).getBoolean());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(1), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(1, v.getElementAtIndex(1).numObjectElements());
+      VariantTestUtil.checkType(
+          v.getElementAtIndex(1).getFieldByKey("key"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      Assert.assertEquals(321, v.getElementAtIndex(1).getFieldByKey("key").getInt());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(2), VariantUtil.PRIMITIVE, Variant.Type.LONG);
+      Assert.assertEquals(1234567890, v.getElementAtIndex(2).getLong());
+
+      VariantTestUtil.checkType(v.getElementAtIndex(3), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Variant nested = v.getElementAtIndex(3);
+      Assert.assertEquals(3, nested.numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(0), VariantUtil.ARRAY, Variant.Type.ARRAY);
+      Assert.assertEquals(0, nested.getElementAtIndex(0).numArrayElements());
+      VariantTestUtil.checkType(nested.getElementAtIndex(1), VariantUtil.SHORT_STR, Variant.Type.STRING);
+      Assert.assertEquals("variant", nested.getElementAtIndex(1).getString());
+      VariantTestUtil.checkType(nested.getElementAtIndex(2), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, nested.getElementAtIndex(2).numObjectElements());
+    });
+  }
+
+  private void buildNested(int i, VariantArrayBuilder obj) {

Review Comment:
   I find this way too hard to reason about for a test. Can you rewrite it to a for loop in `testNestedBuilder`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2070756511


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;

Review Comment:
   Why would this builder support more than one value?
   
   Only object and array builders should support multiple values so I think this builder doesn't need to keep track of this. It can be incremented in the `append` methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2070936092


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset.

Review Comment:
   I think this comment is missing "because it was written last"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069547531


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -451,52 +630,97 @@ public void testNegativeTimestampNtz() {
           (byte) 0xFF,
           (byte) 0xFF
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_NTZ);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_NTZ);
       Assert.assertEquals(
           Instant.parse("1969-12-31T23:59:59.999999Z"), Instant.EPOCH.plus(v.getLong(), ChronoUnit.MICROS));
     });
   }
 
+  @Test
+  public void testTimestampNtzBuilder() {
+    DateTimeFormatter dtf = DateTimeFormatter.ISO_DATE_TIME;
+    VariantBuilder vb = new VariantBuilder(false);
+    long micros = VariantTestUtil.microsSinceEpoch(Instant.from(dtf.parse("2024-01-01T23:00:00.000001Z")));
+    vb.appendTimestampNtz(micros);
+    VariantTestUtil.testVariant(vb.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_NTZ);
+      Assert.assertEquals(micros, v.getLong());
+    });
+  }
+
   @Test
   public void testBinary() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(15), 0x05, 0x00, 0x00, 0x00, 'a', 'b', 'c', 'd', 'e'}),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BINARY);
+        ByteBuffer.wrap(
+            new byte[] {VariantTestUtil.primitiveHeader(15), 0x05, 0x00, 0x00, 0x00, 'a', 'b', 'c', 'd', 'e'
+            }),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BINARY);
       Assert.assertEquals(ByteBuffer.wrap(new byte[] {'a', 'b', 'c', 'd', 'e'}), v.getBinary());
     });
   }
 
+  @Test
+  public void testBinaryBuilder() {
+    VariantBuilder vb = new VariantBuilder(false);
+    byte[] binary = new byte[] {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
+    vb.appendBinary(binary);

Review Comment:
   Shouldn't this append the ByteBuffer instead? Why force the caller to copy into a byte array starting at position 0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2070934029


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,654 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  protected byte[] writeBuffer = new byte[1024];
+
+  protected int writePos = 0;
+  /** The dictionary for mapping keys to monotonically increasing ids. */
+  private final HashMap<String, Integer> dictionary = new HashMap<>();
+  /** The keys in the dictionary, in id order. */
+  private final ArrayList<byte[]> dictionaryKeys = new ArrayList<>();
+
+  /** The number of values appended to this builder. */
+  protected long numValues = 0;
+
+  /**
+   * These are used to build nested objects and arrays, via startObject() and startArray().
+   * Only one of these can be non-null at a time. If one of these is non-null, then no append()
+   * methods can be called on this builder, until endObject() or endArray() is called.
+   */
+  protected VariantObjectBuilder objectBuilder = null;
+
+  protected VariantArrayBuilder arrayBuilder = null;
+
+  /**
+   * Creates a VariantBuilder.
+   */
+  public VariantBuilder() {}
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    if (objectBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an object is being built. Must call endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException(
+          "Cannot call build() while an array is being built. Must call endArray() first.");
+    }
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   */
+  public void appendString(String str) {
+    onAppend();
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HEADER_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    numValues++;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   */
+  public void appendNull() {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HEADER_NULL;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   */
+  public void appendBoolean(boolean b) {
+    onAppend();
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HEADER_TRUE : VariantUtil.HEADER_FALSE;
+    writePos += 1;
+    numValues++;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   */
+  public void appendLong(long l) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   */
+  public void appendInt(int i) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   */
+  public void appendShort(short s) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    numValues++;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   */
+  public void appendByte(byte b) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HEADER_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    numValues++;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   */
+  public void appendDouble(double d) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   */
+  public void appendDecimal(BigDecimal d) {
+    onAppend();
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HEADER_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    numValues++;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   */
+  public void appendDate(int daysSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampTz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   */
+  public void appendTimestampNtz(long microsSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   */
+  public void appendTime(long microsSinceMidnight) {
+    if (microsSinceMidnight < 0) {
+      throw new IllegalArgumentException(
+          String.format("Time value (%d) cannot be negative.", microsSinceMidnight));
+    }
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosTz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   */
+  public void appendTimestampNanosNtz(long nanosSinceEpoch) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HEADER_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    numValues++;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   */
+  public void appendFloat(float f) {
+    onAppend();
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HEADER_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    numValues++;
+  }
+
+  /**
+   * Appends binary data to the variant builder.
+   * @param binary the binary data to append
+   */
+  public void appendBinary(ByteBuffer binary) {
+    onAppend();
+    int binarySize = binary.remaining();
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binarySize);
+    writeBuffer[writePos] = VariantUtil.HEADER_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binarySize, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    ByteBuffer.wrap(writeBuffer, writePos, binarySize).put(binary);
+    writePos += binarySize;
+    numValues++;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   */
+  public void appendUUID(java.util.UUID uuid) {
+    onAppend();
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HEADER_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    numValues++;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder is used
+   * to append object keys and values. startObject() must be called before endObject().
+   * No append*() methods can be called in between startObject() and endObject().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * objBuilder.appendKey("key1");
+   * objBuilder.appendString("value1");
+   * builder.endObject();
+   *
+   * @return a VariantObjectBuilder to build an object
+   */
+  public VariantObjectBuilder startObject() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startObject() without calling endArray() first.");
+    }
+    objectBuilder = new VariantObjectBuilder(this);
+    return objectBuilder;
+  }
+
+  /**
+   * Finishes appending the object to this builder. This method must be called after startObject(),
+   * before other append*() methods can be called on this builder.
+   */
+  protected void endObject() {
+    if (objectBuilder == null) {
+      throw new IllegalStateException("Cannot call endObject() without calling startObject() first.");
+    }
+    ArrayList<FieldEntry> fields = objectBuilder.validateAndGetFields();
+    int numFields = fields.size();
+    Collections.sort(fields);
+    int maxId = numFields == 0 ? 0 : fields.get(0).id;
+    int dataSize = numFields == 0 ? 0 : fields.get(0).valueSize;
+
+    int distinctPos = 0;
+    // Maintain a list of distinct keys in-place.
+    for (int i = 1; i < numFields; ++i) {
+      maxId = Math.max(maxId, fields.get(i).id);
+      if (fields.get(i).id == fields.get(i - 1).id) {
+        // Found a duplicate key. Keep the field with the greater offset.
+        if (fields.get(distinctPos).offset < fields.get(i).offset) {
+          fields.set(distinctPos, fields.get(i));
+        }
+      } else {
+        // Found a distinct key. Add the field to the list.
+        distinctPos++;
+        fields.set(distinctPos, fields.get(i));
+        dataSize += fields.get(i).valueSize;
+      }
+    }
+
+    if (distinctPos + 1 < fields.size()) {
+      numFields = distinctPos + 1;
+      // Resize `fields` to `size`.
+      fields.subList(numFields, fields.size()).clear();
+    }
+
+    boolean largeSize = numFields > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + numFields * idSize + (numFields + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Write the header byte and size entry.
+    writeBuffer[writePos] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, writePos + 1, numFields, sizeBytes);
+
+    int idStart = writePos + 1 + sizeBytes;
+    int offsetStart = idStart + numFields * idSize;
+    int fieldDataStart = offsetStart + (numFields + 1) * offsetSize;
+    int currOffset = 0;
+
+    // Loop over all fields and write the key id, offset, and data to the appropriate offsets
+    for (int i = 0; i < numFields; ++i) {
+      FieldEntry field = fields.get(i);
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, field.id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, currOffset, offsetSize);
+      System.arraycopy(
+          objectBuilder.writeBuffer, field.offset, writeBuffer, fieldDataStart + currOffset, field.valueSize);
+      currOffset += field.valueSize;
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + numFields * offsetSize, dataSize, offsetSize);
+    writePos += headerSize + dataSize;
+    numValues++;
+    objectBuilder = null;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder is used to
+   * append values ot the array. startArray() must be called before endArray(). No append*() methods
+   * can be called in between startArray() and endArray().
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.appendString("value1");
+   * arrayBuilder.appendString("value2");
+   * builder.endArray();
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    onStartNested();
+    if (objectBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endObject() first.");
+    }
+    if (arrayBuilder != null) {
+      throw new IllegalStateException("Cannot call startArray() without calling endArray() first.");
+    }
+    arrayBuilder = new VariantArrayBuilder(this);
+    return arrayBuilder;
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   */
+  public void endArray() {
+    if (arrayBuilder == null) {
+      throw new IllegalStateException("Cannot call endArray() without calling startArray() first.");
+    }
+    ArrayList<Integer> offsets = arrayBuilder.validateAndGetOffsets();
+    int numElements = offsets.size();
+    int dataSize = arrayBuilder.writePos;
+    boolean largeSize = numElements > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (numElements + 1) * offsetSize;
+    checkCapacity(headerSize + dataSize);
+
+    // Copy all the element data to the write buffer.
+    System.arraycopy(arrayBuilder.writeBuffer, 0, writeBuffer, writePos + headerSize, dataSize);

Review Comment:
   Minor: isn't the header just the first byte? I would change the name of `headerSize` to `dataOffset`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#issuecomment-2851837261

   There are a couple of minor comments, but I think overall this is correct and ready -- we can fix exception messages in follow ups and merge this to unblock shredding work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061044970


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);
+    }
+    System.arraycopy(text, 0, writeBuffer, writePos, text.length);
+    writePos += text.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.NULL);
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos++] = VariantUtil.primitiveHeader(b ? VariantUtil.TRUE : VariantUtil.FALSE);
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder. The actual encoded integer type depends on the
+   * value range of the long value.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {

Review Comment:
   This modifies the data as it is written because it suppresses the original type (long). The storage layer should not modify the type of the data passed in so I think this should be `appendLong`, `appendInt`, etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061033687


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetStart + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, stringStart + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetStart + numKeys * offsetSize, currentOffset, offsetSize);
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] text = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = text.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + text.length);
+    if (longStr) {
+      writeBuffer[writePos++] = VariantUtil.primitiveHeader(VariantUtil.LONG_STR);
+      VariantUtil.writeLong(writeBuffer, writePos, text.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos++] = VariantUtil.shortStrHeader(text.length);

Review Comment:
   While it is more succinct, I think it is a bad practice to use the return value of the `++` operator because it forces the reader to remember whether the original value or incremented value is returned along with the context in which that value is used. It is generally more maintainable to use a separate line:
   
   ```java
   writeBuffer[writePos] = VariantUtil.shortStrHeader(text.length);
   writePos += 1;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061029325


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;
+    int stringStart = offsetStart + (numKeys + 1) * offsetSize;
+    long metadataSize = stringStart + dictionaryStringSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {

Review Comment:
   Loop looks correct.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2061016090


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateKeys if true, only the last occurrence of a duplicate key will be kept.
+   *                           Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateKeys) {
+    this.allowDuplicateKeys = allowDuplicateKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryStringSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryStringSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total string size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryStringSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetStart = 1 + offsetSize;

Review Comment:
   I think this could also be more clear as `offsetListOffset`. This is the header plus the dictionary size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069424739


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -329,6 +366,75 @@ static Variant.Type getType(ByteBuffer value) {
     }
   }
 
+  /**
+   * Computes the actual size (in bytes) of the Variant value at `value[pos...]`
+   * @param value The Variant value
+   * @param pos The starting index of the Variant value
+   * @return The size (in bytes) of the Variant value
+   */
+  public static int valueSize(ByteBuffer value, int pos) {

Review Comment:
   I think this can be avoided by a different serialization strategy for arrays and objects because you could use the actual bytes written for the length of elements or field values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069544783


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -137,136 +118,248 @@ public void testNegativeLong() {
           (byte) 0xFF,
           (byte) 0xFF
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
       Assert.assertEquals(-1L, v.getLong());
     });
   }
 
   @Test
   public void testInt() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(1234567890, v.getInt());
     });
   }
 
   @Test
   public void testNegativeInt() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(5), (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF}),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+        ByteBuffer.wrap(new byte[] {
+          VariantTestUtil.primitiveHeader(5), (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF
+        }),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(-1, v.getInt());
     });
   }
 
   @Test
   public void testShort() {
-    Variant value =
-        new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(4), (byte) 0xD2, 0x04}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(4), (byte) 0xD2, 0x04}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
       Assert.assertEquals((short) 1234, v.getShort());
     });
   }
 
   @Test
   public void testNegativeShort() {
-    Variant value =
-        new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(4), (byte) 0xFF, (byte) 0xFF}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(4), (byte) 0xFF, (byte) 0xFF}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
       Assert.assertEquals((short) -1, v.getShort());
     });
   }
 
   @Test
   public void testByte() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(3), 34}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(3), 34}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
       Assert.assertEquals((byte) 34, v.getByte());
     });
   }
 
   @Test
   public void testNegativeByte() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(3), (byte) 0xFF}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(3), (byte) 0xFF}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
       Assert.assertEquals((byte) -1, v.getByte());
     });
   }
 
+  @Test
+  public void testLongBuilder() {
+    Arrays.asList(
+            0L,
+            (long) Byte.MIN_VALUE,
+            (long) Byte.MAX_VALUE,
+            (long) Short.MIN_VALUE,
+            (long) Short.MAX_VALUE,
+            (long) Integer.MIN_VALUE,
+            (long) Integer.MAX_VALUE,
+            Long.MIN_VALUE,
+            Long.MAX_VALUE)
+        .forEach(l -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendLong(l);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
+            Assert.assertEquals((long) l, v.getLong());
+          });
+        });
+  }
+
+  @Test
+  public void testIntBuilder() {
+    Arrays.asList(
+            0,
+            (int) Byte.MIN_VALUE,
+            (int) Byte.MAX_VALUE,
+            (int) Short.MIN_VALUE,
+            (int) Short.MAX_VALUE,
+            Integer.MIN_VALUE,
+            Integer.MAX_VALUE)
+        .forEach(i -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendInt(i);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+            Assert.assertEquals((int) i, v.getInt());
+          });
+        });
+  }
+
+  @Test
+  public void testShortBuilder() {
+    Arrays.asList((short) 0, (short) Byte.MIN_VALUE, (short) Byte.MAX_VALUE, Short.MIN_VALUE, Short.MAX_VALUE)
+        .forEach(s -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendShort(s);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+            Assert.assertEquals((short) s, v.getShort());
+          });
+        });
+  }
+
+  @Test
+  public void testByteBuilder() {

Review Comment:
   Good testing here!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["gene-db (via GitHub)" <gi...@apache.org>]

gene-db commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069712443


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantArrayBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant arrays, used by VariantBuilder.
+ */
+public class VariantArrayBuilder {
+  VariantArrayBuilder(VariantBuilder builder) {
+    this.builder = builder;
+    this.startPos = builder.writePos();
+    this.offsets = new ArrayList<>();
+  }
+
+  void startElement() {
+    offsets.add(builder.writePos() - startPos);
+  }
+
+  int startPos() {
+    return startPos;
+  }
+
+  ArrayList<Integer> offsets() {
+    return offsets;
+  }
+
+  /** The underlying VariantBuilder. */

Review Comment:
   Moved.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -187,6 +187,29 @@ class VariantUtil {
   // The size (in bytes) of a UUID.
   static final int UUID_SIZE = 16;
 
+  // header bytes
+  static final byte HDR_NULL = primitiveHeader(NULL);

Review Comment:
   Renamed.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -137,136 +118,248 @@ public void testNegativeLong() {
           (byte) 0xFF,
           (byte) 0xFF
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
       Assert.assertEquals(-1L, v.getLong());
     });
   }
 
   @Test
   public void testInt() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(1234567890, v.getInt());
     });
   }
 
   @Test
   public void testNegativeInt() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(5), (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF}),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+        ByteBuffer.wrap(new byte[] {
+          VariantTestUtil.primitiveHeader(5), (byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF
+        }),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(-1, v.getInt());
     });
   }
 
   @Test
   public void testShort() {
-    Variant value =
-        new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(4), (byte) 0xD2, 0x04}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(4), (byte) 0xD2, 0x04}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
       Assert.assertEquals((short) 1234, v.getShort());
     });
   }
 
   @Test
   public void testNegativeShort() {
-    Variant value =
-        new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(4), (byte) 0xFF, (byte) 0xFF}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(4), (byte) 0xFF, (byte) 0xFF}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
       Assert.assertEquals((short) -1, v.getShort());
     });
   }
 
   @Test
   public void testByte() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(3), 34}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(3), 34}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
       Assert.assertEquals((byte) 34, v.getByte());
     });
   }
 
   @Test
   public void testNegativeByte() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(3), (byte) 0xFF}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(3), (byte) 0xFF}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
       Assert.assertEquals((byte) -1, v.getByte());
     });
   }
 
+  @Test
+  public void testLongBuilder() {
+    Arrays.asList(
+            0L,
+            (long) Byte.MIN_VALUE,
+            (long) Byte.MAX_VALUE,
+            (long) Short.MIN_VALUE,
+            (long) Short.MAX_VALUE,
+            (long) Integer.MIN_VALUE,
+            (long) Integer.MAX_VALUE,
+            Long.MIN_VALUE,
+            Long.MAX_VALUE)
+        .forEach(l -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendLong(l);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.LONG);
+            Assert.assertEquals((long) l, v.getLong());
+          });
+        });
+  }
+
+  @Test
+  public void testIntBuilder() {
+    Arrays.asList(
+            0,
+            (int) Byte.MIN_VALUE,
+            (int) Byte.MAX_VALUE,
+            (int) Short.MIN_VALUE,
+            (int) Short.MAX_VALUE,
+            Integer.MIN_VALUE,
+            Integer.MAX_VALUE)
+        .forEach(i -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendInt(i);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.INT);
+            Assert.assertEquals((int) i, v.getInt());
+          });
+        });
+  }
+
+  @Test
+  public void testShortBuilder() {
+    Arrays.asList((short) 0, (short) Byte.MIN_VALUE, (short) Byte.MAX_VALUE, Short.MIN_VALUE, Short.MAX_VALUE)
+        .forEach(s -> {
+          VariantBuilder vb2 = new VariantBuilder(false);
+          vb2.appendShort(s);
+          VariantTestUtil.testVariant(vb2.build(), v -> {
+            VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.SHORT);
+            Assert.assertEquals((short) s, v.getShort());
+          });
+        });
+  }
+
+  @Test
+  public void testByteBuilder() {
+    Arrays.asList((byte) 0, Byte.MIN_VALUE, Byte.MAX_VALUE).forEach(b -> {
+      VariantBuilder vb2 = new VariantBuilder(false);
+      vb2.appendByte(b);
+      VariantTestUtil.testVariant(vb2.build(), v -> {
+        VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BYTE);
+        Assert.assertEquals((byte) b, v.getByte());
+      });
+    });
+  }
+
   @Test
   public void testFloat() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(14), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(14), (byte) 0xD2, 0x02, (byte) 0x96, 0x49}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
       Assert.assertEquals(Float.intBitsToFloat(1234567890), v.getFloat(), 0);
     });
   }
 
   @Test
   public void testNegativeFloat() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(14), 0x00, 0x00, 0x00, (byte) 0x80}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(14), 0x00, 0x00, 0x00, (byte) 0x80}),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.FLOAT);
       Assert.assertEquals(-0.0F, v.getFloat(), 0);
     });
   }
 
+  @Test
+  public void testFloatBuilder() {
+    Arrays.asList(Float.MIN_VALUE, 0f, Float.MAX_VALUE).forEach(f -> {

Review Comment:
   Done.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {
+      int distinctPos = 0;
+      // Maintain a list of distinct keys in-place.
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        if (fields.get(i).id == fields.get(i - 1).id) {
+          // Found a duplicate key. Keep the field with the greater offset.
+          if (fields.get(distinctPos).offset < fields.get(i).offset) {
+            fields.set(distinctPos, fields.get(distinctPos).withNewOffset(fields.get(i).offset));
+          }
+        } else {
+          // Found a distinct key. Add the field to the list.
+          ++distinctPos;
+          fields.set(distinctPos, fields.get(i));
+        }
+      }
+      if (distinctPos + 1 < fields.size()) {
+        size = distinctPos + 1;
+        // Resize `fields` to `size`.
+        fields.subList(size, fields.size()).clear();
+        // Sort the fields by offsets so that we can move the value data of each field to the new
+        // offset without overwriting the fields after it.
+        fields.sort(Comparator.comparingInt(f -> f.offset));
+        int currentOffset = 0;
+        for (int i = 0; i < size; ++i) {
+          int oldOffset = fields.get(i).offset;
+          int fieldSize =
+              VariantUtil.valueSize(ByteBuffer.wrap(writeBuffer), objBuilder.startPos() + oldOffset);
+          System.arraycopy(
+              writeBuffer,
+              objBuilder.startPos() + oldOffset,
+              writeBuffer,
+              objBuilder.startPos() + currentOffset,
+              fieldSize);
+          fields.set(i, fields.get(i).withNewOffset(currentOffset));
+          currentOffset += fieldSize;
+        }
+        writePos = objBuilder.startPos() + currentOffset;
+        // Change back to the sort order by field keys, required by the Variant specification.
+        Collections.sort(fields);
+      }
+    } else {
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        String key = fields.get(i).key;
+        if (key.equals(fields.get(i - 1).key)) {
+          throw new IllegalStateException("Failed to build Variant because of duplicate object key: " + key);
+        }
+      }
+    }
+    int dataSize = writePos - objBuilder.startPos();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + size * idSize + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the object header section.
+    System.arraycopy(writeBuffer, objBuilder.startPos(), writeBuffer, objBuilder.startPos() + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[objBuilder.startPos()] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, objBuilder.startPos() + 1, size, sizeBytes);
+    int idStart = objBuilder.startPos() + 1 + sizeBytes;
+    int offsetStart = idStart + size * idSize;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, fields.get(i).id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, fields.get(i).offset, offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder must be
+   * used for future calls to startArrayElement() and endArray(). To add an element to the array,
+   * call startArrayElement() and then append the value. endArray() must be called to finish writing
+   * the Variant array.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.startArrayElement(arrayBuilder);
+   * builder.appendString("value1");
+   * arrayBuilder.endArray(arrayBuilder);
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    return new VariantArrayBuilder(this);
+  }
+
+  /**
+   * Starts appending an element to the array. This method must be called before appending the
+   * corresponding value.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   */
+  public void startArrayElement(VariantArrayBuilder arrayBuilder) {
+    arrayBuilder.startElement();
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endArray(VariantArrayBuilder arrayBuilder) {

Review Comment:
   Updated the array builder api.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObject.java:
##########
@@ -205,22 +137,52 @@ private static ByteBuffer constructMetadata(Boolean isSorted, List<String> field
 
   @Test
   public void testEmptyObject() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {0b10, 0x00}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    Variant value = new Variant(ByteBuffer.wrap(new byte[] {0b10, 0x00}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(0, v.numObjectElements());
+    });
+  }
+
+  @Test
+  public void testEmptyObjectBuilder() {
+    VariantBuilder b = new VariantBuilder(true);
+    VariantObjectBuilder o = b.startObject();
+    b.endObject(o);
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(0, v.numObjectElements());
     });
   }
 
   @Test
   public void testEmptyLargeObject() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {0b1000010, 0x00, 0x00, 0x00, 0x00}), EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {0b1000010, 0x00, 0x00, 0x00, 0x00}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(0, v.numObjectElements());
     });
   }
 
+  @Test
+  public void testLargeObjectBuilder() {
+    VariantBuilder b = new VariantBuilder(true);
+    VariantObjectBuilder o = b.startObject();
+    for (int i = 0; i < 1234; i++) {
+      b.appendObjectKey(o, "a" + i);
+      b.appendLong(i);
+    }
+    b.endObject(o);

Review Comment:
   I added checks to avoid invalid usage of the api. The `appendKey()` and `appendValue()` must alternate.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -34,91 +37,69 @@
 public class TestVariantScalar {
   private static final Logger LOG = LoggerFactory.getLogger(TestVariantScalar.class);
 
-  private static final ByteBuffer EMPTY_METADATA = ByteBuffer.wrap(new byte[] {0b1});
-
-  private void checkType(Variant v, int expectedBasicType, Variant.Type expectedType) {
-    Assert.assertEquals(expectedBasicType, v.value.get(v.value.position()) & VariantUtil.BASIC_TYPE_MASK);
-    Assert.assertEquals(expectedType, v.getType());
-  }
-
-  private void testVariant(Variant v, Consumer<Variant> consumer) {
-    consumer.accept(v);
-    // Create new Variant with different byte offsets
-    byte[] newValue = new byte[v.value.capacity() + 50];
-    byte[] newMetadata = new byte[v.metadata.capacity() + 50];
-    Arrays.fill(newValue, (byte) 0xFF);
-    Arrays.fill(newMetadata, (byte) 0xFF);
-    v.value.position(0);
-    v.value.get(newValue, 25, v.value.capacity());
-    v.value.position(0);
-    v.metadata.position(0);
-    v.metadata.get(newMetadata, 25, v.metadata.capacity());
-    v.metadata.position(0);
-    Variant v2 = new Variant(
-        ByteBuffer.wrap(newValue, 25, v.value.capacity()),
-        ByteBuffer.wrap(newMetadata, 25, v.metadata.capacity()));
-    consumer.accept(v2);
-  }
-
-  private static byte primitiveHeader(int type) {
-    return (byte) (type << 2);
-  }
-
-  private static byte metadataHeader(boolean isSorted, int offsetSize) {
-    return (byte) (((offsetSize - 1) << 6) | (isSorted ? 0b10000 : 0) | 0b0001);
-  }
-
-  private static int getMinIntegerSize(int value) {
-    return (value <= 0xFF) ? 1 : (value <= 0xFFFF) ? 2 : (value <= 0xFFFFFF) ? 3 : 4;
-  }
-
-  private static void writeVarlenInt(ByteBuffer buffer, int value, int valueSize) {
-    if (valueSize == 1) {
-      buffer.put((byte) value);
-    } else if (valueSize == 2) {
-      buffer.putShort((short) value);
-    } else if (valueSize == 3) {
-      buffer.put((byte) (value & 0xFF));
-      buffer.put((byte) ((value >> 8) & 0xFF));
-      buffer.put((byte) ((value >> 16) & 0xFF));
-    } else {
-      buffer.putInt(value);
-    }
+  @Test
+  public void testNull() {
+    Variant value = new Variant(
+        ByteBuffer.wrap(new byte[] {VariantTestUtil.primitiveHeader(0)}), VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> VariantTestUtil.checkType(v, VariantUtil.NULL, Variant.Type.NULL));
   }
 
   @Test
-  public void testNull() {
-    Variant value = new Variant(ByteBuffer.wrap(new byte[] {primitiveHeader(0)}), EMPTY_METADATA);
-    testVariant(value, v -> checkType(v, VariantUtil.NULL, Variant.Type.NULL));
+  public void testNullBuilder() {
+    VariantBuilder vb = new VariantBuilder(false);
+    vb.appendNull();

Review Comment:
   Added checks to disallow multiple appends.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantArrayBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant arrays, used by VariantBuilder.

Review Comment:
   We decided to do that separately.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObject.java:
##########
@@ -250,22 +212,61 @@ public void testMixedObject() {
     Variant value = new Variant(
         ByteBuffer.wrap(constructObject(keys, fields, true)),
         constructMetadata(true, ImmutableList.of("a", "b", "c")));
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(3, v.numObjectElements());
-      checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      VariantTestUtil.checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(1234567890, v.getFieldByKey("a").getInt());
-      checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      VariantTestUtil.checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
       Assert.assertTrue(v.getFieldByKey("b").getBoolean());
-      checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      VariantTestUtil.checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
 
       Variant nestedV = v.getFieldByKey("c");
       Assert.assertEquals(2, nestedV.numObjectElements());
-      checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
       Assert.assertEquals(
           LocalDate.parse("2025-04-17"),
           LocalDate.ofEpochDay(nestedV.getFieldByKey("a").getInt()));
-      checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+    });
+  }
+
+  @Test
+  public void testMixedObjectBuilder() {

Review Comment:
   Separated the builder tests to their own files.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantUtil.java:
##########
@@ -329,6 +366,75 @@ static Variant.Type getType(ByteBuffer value) {
     }
   }
 
+  /**
+   * Computes the actual size (in bytes) of the Variant value at `value[pos...]`
+   * @param value The Variant value
+   * @param pos The starting index of the Variant value
+   * @return The size (in bytes) of the Variant value
+   */
+  public static int valueSize(ByteBuffer value, int pos) {

Review Comment:
   Removed.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {

Review Comment:
   Ok, sounds good. Updated to handle the duplicates and removed the flag.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/VariantTestUtil.java:
##########
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.security.SecureRandom;
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import org.junit.Assert;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class VariantTestUtil {
+  private static final Logger LOG = LoggerFactory.getLogger(VariantTestUtil.class);
+  private static final String RANDOM_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
+
+  /** Random number generator for generating random strings */
+  private static SecureRandom random = new SecureRandom(new byte[] {1, 2, 3, 4, 5});
+
+  static final ByteBuffer EMPTY_METADATA = ByteBuffer.wrap(new byte[] {0b1});
+
+  static void checkType(Variant v, int expectedBasicType, Variant.Type expectedType) {
+    Assert.assertEquals(expectedBasicType, v.value.get(v.value.position()) & VariantUtil.BASIC_TYPE_MASK);
+    Assert.assertEquals(expectedType, v.getType());
+  }
+
+  static long microsSinceEpoch(Instant instant) {

Review Comment:
   Sounds good, removed these micro and nano utilities, and just check the literal values in the tests.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -415,23 +574,43 @@ public void testNegativeTimestamp() {
           (byte) 0xFF,
           (byte) 0xFF
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_TZ);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_TZ);
       Assert.assertEquals(
           Instant.parse("1969-12-31T23:59:59.999999Z"), Instant.EPOCH.plus(v.getLong(), ChronoUnit.MICROS));
     });
   }
 
+  @Test
+  public void testTimestampTzBuilder() {
+    DateTimeFormatter dtf = DateTimeFormatter.ISO_DATE_TIME;
+    VariantBuilder vb = new VariantBuilder(false);
+    long micros = VariantTestUtil.microsSinceEpoch(Instant.from(dtf.parse("2024-12-16T10:23:45.321456-08:00")));

Review Comment:
   Removed the parsing.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {
+      int distinctPos = 0;
+      // Maintain a list of distinct keys in-place.
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        if (fields.get(i).id == fields.get(i - 1).id) {
+          // Found a duplicate key. Keep the field with the greater offset.
+          if (fields.get(distinctPos).offset < fields.get(i).offset) {
+            fields.set(distinctPos, fields.get(distinctPos).withNewOffset(fields.get(i).offset));
+          }
+        } else {
+          // Found a distinct key. Add the field to the list.
+          ++distinctPos;
+          fields.set(distinctPos, fields.get(i));
+        }
+      }
+      if (distinctPos + 1 < fields.size()) {
+        size = distinctPos + 1;
+        // Resize `fields` to `size`.
+        fields.subList(size, fields.size()).clear();
+        // Sort the fields by offsets so that we can move the value data of each field to the new
+        // offset without overwriting the fields after it.
+        fields.sort(Comparator.comparingInt(f -> f.offset));
+        int currentOffset = 0;
+        for (int i = 0; i < size; ++i) {
+          int oldOffset = fields.get(i).offset;
+          int fieldSize =
+              VariantUtil.valueSize(ByteBuffer.wrap(writeBuffer), objBuilder.startPos() + oldOffset);
+          System.arraycopy(
+              writeBuffer,
+              objBuilder.startPos() + oldOffset,
+              writeBuffer,
+              objBuilder.startPos() + currentOffset,
+              fieldSize);
+          fields.set(i, fields.get(i).withNewOffset(currentOffset));
+          currentOffset += fieldSize;
+        }
+        writePos = objBuilder.startPos() + currentOffset;
+        // Change back to the sort order by field keys, required by the Variant specification.
+        Collections.sort(fields);
+      }
+    } else {
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        String key = fields.get(i).key;
+        if (key.equals(fields.get(i - 1).key)) {
+          throw new IllegalStateException("Failed to build Variant because of duplicate object key: " + key);
+        }
+      }
+    }
+    int dataSize = writePos - objBuilder.startPos();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + size * idSize + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the object header section.
+    System.arraycopy(writeBuffer, objBuilder.startPos(), writeBuffer, objBuilder.startPos() + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[objBuilder.startPos()] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, objBuilder.startPos() + 1, size, sizeBytes);
+    int idStart = objBuilder.startPos() + 1 + sizeBytes;
+    int offsetStart = idStart + size * idSize;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, fields.get(i).id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, fields.get(i).offset, offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder must be
+   * used for future calls to startArrayElement() and endArray(). To add an element to the array,
+   * call startArrayElement() and then append the value. endArray() must be called to finish writing
+   * the Variant array.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.startArrayElement(arrayBuilder);
+   * builder.appendString("value1");
+   * arrayBuilder.endArray(arrayBuilder);
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    return new VariantArrayBuilder(this);
+  }
+
+  /**
+   * Starts appending an element to the array. This method must be called before appending the
+   * corresponding value.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   */
+  public void startArrayElement(VariantArrayBuilder arrayBuilder) {
+    arrayBuilder.startElement();
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endArray(VariantArrayBuilder arrayBuilder) {
+    int start = arrayBuilder.startPos();
+    int dataSize = writePos - start;
+    int size = arrayBuilder.offsets().size();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the header section.
+    System.arraycopy(writeBuffer, start, writeBuffer, start + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[start] = VariantUtil.arrayHeader(largeSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, start + 1, size, sizeBytes);
+    int offsetStart = start + 1 + sizeBytes;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(
+          writeBuffer,
+          offsetStart + i * offsetSize,
+          arrayBuilder.offsets().get(i),
+          offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Adds a key to the Variant dictionary. If the key already exists, the dictionary is unmodified.
+   * @param key the key to add
+   * @return the id of the key
+   */
+  int addKey(String key) {
+    return dictionary.computeIfAbsent(key, newKey -> {
+      int id = dictionaryKeys.size();
+      dictionaryKeys.add(newKey.getBytes(StandardCharsets.UTF_8));
+      return id;
+    });
+  }
+
+  /**
+   * @return the current write position of the variant builder
+   */
+  int writePos() {
+    return writePos;
+  }
+
+  /**
+   * Class to store the information of a Variant object field. We need to collect all fields of
+   * an object, sort them by their keys, and build the Variant object in sorted order.
+   */
+  static final class FieldEntry implements Comparable<FieldEntry> {
+    final String key;
+    final int id;
+    final int offset;
+
+    FieldEntry(String key, int id, int offset) {
+      this.key = key;
+      this.id = id;
+      this.offset = offset;
+    }
+
+    FieldEntry withNewOffset(int newOffset) {
+      return new FieldEntry(key, id, newOffset);
+    }
+
+    @Override
+    public int compareTo(FieldEntry other) {
+      return key.compareTo(other.key);
+    }
+  }
+
+  private void checkCapacity(int additionalBytes) {
+    int requiredBytes = writePos + additionalBytes;
+    if (requiredBytes > writeBuffer.length) {
+      // Allocate a new buffer with a capacity of the next power of 2 of `requiredBytes`.
+      int newCapacity = Integer.highestOneBit(requiredBytes);
+      newCapacity = newCapacity < requiredBytes ? newCapacity * 2 : newCapacity;
+      byte[] newValue = new byte[newCapacity];
+      System.arraycopy(writeBuffer, 0, newValue, 0, writePos);
+      writeBuffer = newValue;
+    }
+  }
+
+  private int getMinIntegerSize(int value) {
+    assert value >= 0;
+    if (value <= VariantUtil.U8_MAX) {
+      return VariantUtil.U8_SIZE;
+    }
+    if (value <= VariantUtil.U16_MAX) {
+      return VariantUtil.U16_SIZE;
+    }
+    if (value <= VariantUtil.U24_MAX) {
+      return VariantUtil.U24_SIZE;
+    }
+    return VariantUtil.U32_SIZE;
+  }
+
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  private byte[] writeBuffer = new byte[128];
+
+  private int writePos = 0;

Review Comment:
   Moved to top.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);

Review Comment:
   Updated the API so that the object builder writes to its own buffer, and will do the key/value validation.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantBuilder.java:
##########
@@ -0,0 +1,621 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+
+/**
+ * Builder for creating Variant value and metadata.
+ */
+public class VariantBuilder {
+  /**
+   * Creates a VariantBuilder.
+   * @param allowDuplicateObjectKeys if true, only the last occurrence of a duplicate object key
+   *                                 will be kept. Otherwise, an exception will be thrown.
+   */
+  public VariantBuilder(boolean allowDuplicateObjectKeys) {
+    this.allowDuplicateObjectKeys = allowDuplicateObjectKeys;
+  }
+
+  /**
+   * @return the Variant value
+   */
+  public Variant build() {
+    int numKeys = dictionaryKeys.size();
+    // Use long to avoid overflow in accumulating lengths.
+    long dictionaryTotalDataSize = 0;
+    for (byte[] key : dictionaryKeys) {
+      dictionaryTotalDataSize += key.length;
+    }
+    // Determine the number of bytes required per offset entry.
+    // The largest offset is the one-past-the-end value, which is total data size. It's very
+    // unlikely that the number of keys could be larger, but incorporate that into the calculation
+    // in case of pathological data.
+    long maxSize = Math.max(dictionaryTotalDataSize, numKeys);
+    int offsetSize = getMinIntegerSize((int) maxSize);
+
+    int offsetListOffset = 1 + offsetSize;
+    int dataOffset = offsetListOffset + (numKeys + 1) * offsetSize;
+    long metadataSize = dataOffset + dictionaryTotalDataSize;
+
+    byte[] metadata = new byte[(int) metadataSize];
+    // Only unsorted dictionary keys are supported.
+    // TODO: Support sorted dictionary keys.
+    int headerByte = VariantUtil.VERSION | ((offsetSize - 1) << 6);
+    VariantUtil.writeLong(metadata, 0, headerByte, 1);
+    VariantUtil.writeLong(metadata, 1, numKeys, offsetSize);
+    int currentOffset = 0;
+    for (int i = 0; i < numKeys; ++i) {
+      VariantUtil.writeLong(metadata, offsetListOffset + i * offsetSize, currentOffset, offsetSize);
+      byte[] key = dictionaryKeys.get(i);
+      System.arraycopy(key, 0, metadata, dataOffset + currentOffset, key.length);
+      currentOffset += key.length;
+    }
+    VariantUtil.writeLong(metadata, offsetListOffset + numKeys * offsetSize, currentOffset, offsetSize);
+    // Copying the data to a new buffer, to retain only the required data length, not the capacity.
+    return new Variant(Arrays.copyOfRange(writeBuffer, 0, writePos), metadata);
+  }
+
+  /**
+   * Appends a string value to the Variant builder.
+   * @param str the string value to append
+   * @return this builder
+   */
+  public VariantBuilder appendString(String str) {
+    byte[] data = str.getBytes(StandardCharsets.UTF_8);
+    boolean longStr = data.length > VariantUtil.MAX_SHORT_STR_SIZE;
+    checkCapacity((longStr ? 1 + VariantUtil.U32_SIZE : 1) + data.length);
+    if (longStr) {
+      writeBuffer[writePos] = VariantUtil.HDR_LONG_STRING;
+      writePos += 1;
+      VariantUtil.writeLong(writeBuffer, writePos, data.length, VariantUtil.U32_SIZE);
+      writePos += VariantUtil.U32_SIZE;
+    } else {
+      writeBuffer[writePos] = VariantUtil.shortStrHeader(data.length);
+      writePos += 1;
+    }
+    System.arraycopy(data, 0, writeBuffer, writePos, data.length);
+    writePos += data.length;
+    return this;
+  }
+
+  /**
+   * Appends a null value to the Variant builder.
+   * @return this builder
+   */
+  public VariantBuilder appendNull() {
+    checkCapacity(1);
+    writeBuffer[writePos] = VariantUtil.HDR_NULL;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a boolean value to the Variant builder.
+   * @param b the boolean value to append
+   * @return this builder
+   */
+  public VariantBuilder appendBoolean(boolean b) {
+    checkCapacity(1);
+    writeBuffer[writePos] = b ? VariantUtil.HDR_TRUE : VariantUtil.HDR_FALSE;
+    writePos += 1;
+    return this;
+  }
+
+  /**
+   * Appends a long value to the variant builder.
+   * @param l the long value to append
+   * @return this builder
+   */
+  public VariantBuilder appendLong(long l) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_INT64;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, l, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends an int value to the variant builder.
+   * @param i the int to append
+   * @return this builder
+   */
+  public VariantBuilder appendInt(int i) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_INT32;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, i, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a short value to the variant builder.
+   * @param s the short to append
+   * @return this builder
+   */
+  public VariantBuilder appendShort(short s) {
+    checkCapacity(1 /* header size */ + 2);
+    writeBuffer[writePos] = VariantUtil.HDR_INT16;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, s, 2);
+    writePos += 3;
+    return this;
+  }
+
+  /**
+   * Appends a byte value to the variant builder.
+   * @param b the byte to append
+   * @return this builder
+   */
+  public VariantBuilder appendByte(byte b) {
+    checkCapacity(1 /* header size */ + 1);
+    writeBuffer[writePos] = VariantUtil.HDR_INT8;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, b, 1);
+    writePos += 2;
+    return this;
+  }
+
+  /**
+   * Appends a double value to the variant builder.
+   * @param d the double to append
+   * @return this builder
+   */
+  public VariantBuilder appendDouble(double d) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_DOUBLE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Double.doubleToLongBits(d), 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a decimal value to the variant builder. The actual encoded decimal type depends on the
+   * precision and scale of the decimal value.
+   * @param d the decimal value to append
+   * @return this builder
+   */
+  public VariantBuilder appendDecimal(BigDecimal d) {
+    BigInteger unscaled = d.unscaledValue();
+    if (d.scale() <= VariantUtil.MAX_DECIMAL4_PRECISION && d.precision() <= VariantUtil.MAX_DECIMAL4_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 4);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL4;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.intValueExact(), 4);
+      writePos += 6;
+    } else if (d.scale() <= VariantUtil.MAX_DECIMAL8_PRECISION
+        && d.precision() <= VariantUtil.MAX_DECIMAL8_PRECISION) {
+      checkCapacity(2 /* header and scale size */ + 8);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL8;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      VariantUtil.writeLong(writeBuffer, writePos + 2, unscaled.longValueExact(), 8);
+      writePos += 10;
+    } else {
+      assert d.scale() <= VariantUtil.MAX_DECIMAL16_PRECISION
+          && d.precision() <= VariantUtil.MAX_DECIMAL16_PRECISION;
+      checkCapacity(2 /* header and scale size */ + 16);
+      writeBuffer[writePos] = VariantUtil.HDR_DECIMAL16;
+      writeBuffer[writePos + 1] = (byte) d.scale();
+      writePos += 2;
+      // `toByteArray` returns a big-endian representation. We need to copy it reversely and sign
+      // extend it to 16 bytes.
+      byte[] bytes = unscaled.toByteArray();
+      for (int i = 0; i < bytes.length; ++i) {
+        writeBuffer[writePos + i] = bytes[bytes.length - 1 - i];
+      }
+      byte sign = (byte) (bytes[0] < 0 ? -1 : 0);
+      for (int i = bytes.length; i < 16; ++i) {
+        writeBuffer[writePos + i] = sign;
+      }
+      writePos += 16;
+    }
+    return this;
+  }
+
+  /**
+   * Appends a date value to the variant builder. The date is represented as the number of days
+   * since the epoch.
+   * @param daysSinceEpoch the number of days since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendDate(int daysSinceEpoch) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_DATE;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, daysSinceEpoch, 4);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampTz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampTz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNtz value to the variant builder. The timestamp is represented as the number
+   * of microseconds since the epoch.
+   * @param microsSinceEpoch the number of microseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNtz(long microsSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a Time value to the variant builder. The time is represented as the number of
+   * microseconds since midnight.
+   * @param microsSinceMidnight the number of microseconds since midnight
+   * @return this builder
+   */
+  public VariantBuilder appendTime(long microsSinceMidnight) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIME;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, microsSinceMidnight, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosTz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosTz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_TZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a TimestampNanosNtz value to the variant builder. The timestamp is represented as the
+   * number of nanoseconds since the epoch.
+   * @param nanosSinceEpoch the number of nanoseconds since the epoch
+   * @return this builder
+   */
+  public VariantBuilder appendTimestampNanosNtz(long nanosSinceEpoch) {
+    checkCapacity(1 /* header size */ + 8);
+    writeBuffer[writePos] = VariantUtil.HDR_TIMESTAMP_NANOS_NTZ;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, nanosSinceEpoch, 8);
+    writePos += 9;
+    return this;
+  }
+
+  /**
+   * Appends a float value to the variant builder.
+   * @param f the float to append
+   * @return this builder
+   */
+  public VariantBuilder appendFloat(float f) {
+    checkCapacity(1 /* header size */ + 4);
+    writeBuffer[writePos] = VariantUtil.HDR_FLOAT;
+    VariantUtil.writeLong(writeBuffer, writePos + 1, Float.floatToIntBits(f), 8);
+    writePos += 5;
+    return this;
+  }
+
+  /**
+   * Appends a byte array to the variant builder.
+   * @param binary the byte array to append
+   * @return this builder
+   */
+  public VariantBuilder appendBinary(byte[] binary) {
+    checkCapacity(1 /* header size */ + VariantUtil.U32_SIZE + binary.length);
+    writeBuffer[writePos] = VariantUtil.HDR_BINARY;
+    writePos += 1;
+    VariantUtil.writeLong(writeBuffer, writePos, binary.length, VariantUtil.U32_SIZE);
+    writePos += VariantUtil.U32_SIZE;
+    System.arraycopy(binary, 0, writeBuffer, writePos, binary.length);
+    writePos += binary.length;
+    return this;
+  }
+
+  /**
+   * Appends a UUID value to the variant builder.
+   * @param uuid the UUID to append
+   * @return this builder
+   */
+  public VariantBuilder appendUUID(java.util.UUID uuid) {
+    checkCapacity(1 /* header size */ + VariantUtil.UUID_SIZE);
+    writeBuffer[writePos] = VariantUtil.HDR_UUID;
+    writePos += 1;
+
+    ByteBuffer bb =
+        ByteBuffer.wrap(writeBuffer, writePos, VariantUtil.UUID_SIZE).order(ByteOrder.BIG_ENDIAN);
+    bb.putLong(uuid.getMostSignificantBits());
+    bb.putLong(uuid.getLeastSignificantBits());
+    writePos += VariantUtil.UUID_SIZE;
+    return this;
+  }
+
+  /**
+   * Starts appending an object to this variant builder. The returned VariantObjectBuilder must
+   * be used for future calls to addObjectKey() and endObject(). To add a (key, value) to the
+   * Variant object, call appendObjectKey() to add the key name, and then append the value.
+   * endObject() must be called to finish writing the Variant object.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantObjectBuilder objBuilder = builder.startObject();
+   * builder.appendObjectKey(objBuilder, "key1");
+   * builder.appendString("value1");
+   * builder.endObject(objBuilder);
+   *
+   * @return a VariantObjectBuilder to use for appendObjectKey() and endObject().
+   */
+  public VariantObjectBuilder startObject() {
+    return new VariantObjectBuilder(this);
+  }
+
+  /**
+   * Appends a key to the Variant object. This method must be called before appending the
+   * corresponding value.
+   * @param objBuilder the VariantObjectBuilder to use
+   */
+  public void appendObjectKey(VariantObjectBuilder objBuilder, String key) {
+    objBuilder.appendKey(key);
+  }
+
+  /**
+   * Ends appending an object to this variant builder. This method must be called after all keys and
+   * values have been added to the object.
+   * @param objBuilder the VariantObjectBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endObject(VariantObjectBuilder objBuilder) {
+    ArrayList<FieldEntry> fields = objBuilder.fields();
+    int size = fields.size();
+    Collections.sort(fields);
+    int maxId = size == 0 ? 0 : fields.get(0).id;
+    if (allowDuplicateObjectKeys) {
+      int distinctPos = 0;
+      // Maintain a list of distinct keys in-place.
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        if (fields.get(i).id == fields.get(i - 1).id) {
+          // Found a duplicate key. Keep the field with the greater offset.
+          if (fields.get(distinctPos).offset < fields.get(i).offset) {
+            fields.set(distinctPos, fields.get(distinctPos).withNewOffset(fields.get(i).offset));
+          }
+        } else {
+          // Found a distinct key. Add the field to the list.
+          ++distinctPos;
+          fields.set(distinctPos, fields.get(i));
+        }
+      }
+      if (distinctPos + 1 < fields.size()) {
+        size = distinctPos + 1;
+        // Resize `fields` to `size`.
+        fields.subList(size, fields.size()).clear();
+        // Sort the fields by offsets so that we can move the value data of each field to the new
+        // offset without overwriting the fields after it.
+        fields.sort(Comparator.comparingInt(f -> f.offset));
+        int currentOffset = 0;
+        for (int i = 0; i < size; ++i) {
+          int oldOffset = fields.get(i).offset;
+          int fieldSize =
+              VariantUtil.valueSize(ByteBuffer.wrap(writeBuffer), objBuilder.startPos() + oldOffset);
+          System.arraycopy(
+              writeBuffer,
+              objBuilder.startPos() + oldOffset,
+              writeBuffer,
+              objBuilder.startPos() + currentOffset,
+              fieldSize);
+          fields.set(i, fields.get(i).withNewOffset(currentOffset));
+          currentOffset += fieldSize;
+        }
+        writePos = objBuilder.startPos() + currentOffset;
+        // Change back to the sort order by field keys, required by the Variant specification.
+        Collections.sort(fields);
+      }
+    } else {
+      for (int i = 1; i < size; ++i) {
+        maxId = Math.max(maxId, fields.get(i).id);
+        String key = fields.get(i).key;
+        if (key.equals(fields.get(i - 1).key)) {
+          throw new IllegalStateException("Failed to build Variant because of duplicate object key: " + key);
+        }
+      }
+    }
+    int dataSize = writePos - objBuilder.startPos();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int idSize = getMinIntegerSize(maxId);
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, id list, and offset list.
+    int headerSize = 1 + sizeBytes + size * idSize + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the object header section.
+    System.arraycopy(writeBuffer, objBuilder.startPos(), writeBuffer, objBuilder.startPos() + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[objBuilder.startPos()] = VariantUtil.objectHeader(largeSize, idSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, objBuilder.startPos() + 1, size, sizeBytes);
+    int idStart = objBuilder.startPos() + 1 + sizeBytes;
+    int offsetStart = idStart + size * idSize;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(writeBuffer, idStart + i * idSize, fields.get(i).id, idSize);
+      VariantUtil.writeLong(writeBuffer, offsetStart + i * offsetSize, fields.get(i).offset, offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Starts appending an array to this variant builder. The returned VariantArrayBuilder must be
+   * used for future calls to startArrayElement() and endArray(). To add an element to the array,
+   * call startArrayElement() and then append the value. endArray() must be called to finish writing
+   * the Variant array.
+   *
+   * Example usage:
+   * VariantBuilder builder = new VariantBuilder();
+   * VariantArrayBuilder arrayBuilder = builder.startArray();
+   * arrayBuilder.startArrayElement(arrayBuilder);
+   * builder.appendString("value1");
+   * arrayBuilder.endArray(arrayBuilder);
+   *
+   * @return a VariantArrayBuilder to use for startArrayElement() and endArray().
+   */
+  public VariantArrayBuilder startArray() {
+    return new VariantArrayBuilder(this);
+  }
+
+  /**
+   * Starts appending an element to the array. This method must be called before appending the
+   * corresponding value.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   */
+  public void startArrayElement(VariantArrayBuilder arrayBuilder) {
+    arrayBuilder.startElement();
+  }
+
+  /**
+   * Ends appending an array to this variant builder. This method must be called after all elements
+   * have been added to the array.
+   * @param arrayBuilder the VariantArrayBuilder to use
+   * @return this builder
+   */
+  public VariantBuilder endArray(VariantArrayBuilder arrayBuilder) {
+    int start = arrayBuilder.startPos();
+    int dataSize = writePos - start;
+    int size = arrayBuilder.offsets().size();
+    boolean largeSize = size > VariantUtil.U8_MAX;
+    int sizeBytes = largeSize ? VariantUtil.U32_SIZE : 1;
+    int offsetSize = getMinIntegerSize(dataSize);
+    // The space for header byte, object size, and offset list.
+    int headerSize = 1 + sizeBytes + (size + 1) * offsetSize;
+    checkCapacity(headerSize);
+    // Shift the just-written field data to make room for the header section.
+    System.arraycopy(writeBuffer, start, writeBuffer, start + headerSize, dataSize);
+    writePos += headerSize;
+    writeBuffer[start] = VariantUtil.arrayHeader(largeSize, offsetSize);
+    VariantUtil.writeLong(writeBuffer, start + 1, size, sizeBytes);
+    int offsetStart = start + 1 + sizeBytes;
+    for (int i = 0; i < size; ++i) {
+      VariantUtil.writeLong(
+          writeBuffer,
+          offsetStart + i * offsetSize,
+          arrayBuilder.offsets().get(i),
+          offsetSize);
+    }
+    VariantUtil.writeLong(writeBuffer, offsetStart + size * offsetSize, dataSize, offsetSize);
+    return this;
+  }
+
+  /**
+   * Adds a key to the Variant dictionary. If the key already exists, the dictionary is unmodified.
+   * @param key the key to add
+   * @return the id of the key
+   */
+  int addKey(String key) {
+    return dictionary.computeIfAbsent(key, newKey -> {
+      int id = dictionaryKeys.size();
+      dictionaryKeys.add(newKey.getBytes(StandardCharsets.UTF_8));
+      return id;
+    });
+  }
+
+  /**
+   * @return the current write position of the variant builder
+   */
+  int writePos() {
+    return writePos;
+  }
+
+  /**
+   * Class to store the information of a Variant object field. We need to collect all fields of
+   * an object, sort them by their keys, and build the Variant object in sorted order.
+   */
+  static final class FieldEntry implements Comparable<FieldEntry> {
+    final String key;
+    final int id;
+    final int offset;
+
+    FieldEntry(String key, int id, int offset) {
+      this.key = key;
+      this.id = id;
+      this.offset = offset;
+    }
+
+    FieldEntry withNewOffset(int newOffset) {
+      return new FieldEntry(key, id, newOffset);
+    }
+
+    @Override
+    public int compareTo(FieldEntry other) {
+      return key.compareTo(other.key);
+    }
+  }
+
+  private void checkCapacity(int additionalBytes) {
+    int requiredBytes = writePos + additionalBytes;
+    if (requiredBytes > writeBuffer.length) {
+      // Allocate a new buffer with a capacity of the next power of 2 of `requiredBytes`.
+      int newCapacity = Integer.highestOneBit(requiredBytes);
+      newCapacity = newCapacity < requiredBytes ? newCapacity * 2 : newCapacity;
+      byte[] newValue = new byte[newCapacity];
+      System.arraycopy(writeBuffer, 0, newValue, 0, writePos);
+      writeBuffer = newValue;
+    }
+  }
+
+  private int getMinIntegerSize(int value) {
+    assert value >= 0;
+    if (value <= VariantUtil.U8_MAX) {
+      return VariantUtil.U8_SIZE;
+    }
+    if (value <= VariantUtil.U16_MAX) {
+      return VariantUtil.U16_SIZE;
+    }
+    if (value <= VariantUtil.U24_MAX) {
+      return VariantUtil.U24_SIZE;
+    }
+    return VariantUtil.U32_SIZE;
+  }
+
+  /** The buffer for building the Variant value. The first `writePos` bytes have been written. */
+  private byte[] writeBuffer = new byte[128];

Review Comment:
   Updated to 1k.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantArray.java:
##########
@@ -20,104 +20,32 @@
 
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
-import java.nio.charset.StandardCharsets;
-import java.security.SecureRandom;
 import java.time.LocalDate;
-import java.util.*;
-import java.util.function.Consumer;
 import org.junit.Assert;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class TestVariantArray {
   private static final Logger LOG = LoggerFactory.getLogger(TestVariantArray.class);
-  private static final String RANDOM_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
 
-  /** Random number generator for generating random strings */
-  private static SecureRandom random = new SecureRandom(new byte[] {1, 2, 3, 4, 5});
-
-  private static final ByteBuffer EMPTY_METADATA = ByteBuffer.wrap(new byte[] {0b1});
-
-  private static final byte[] VALUE_NULL = new byte[] {primitiveHeader(0)};
-  private static final byte[] VALUE_BOOL = new byte[] {primitiveHeader(1)};
-  private static final byte[] VALUE_INT = new byte[] {primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49};
+  private static final byte[] VALUE_NULL = new byte[] {VariantTestUtil.primitiveHeader(0)};
+  private static final byte[] VALUE_BOOL = new byte[] {VariantTestUtil.primitiveHeader(1)};
+  private static final byte[] VALUE_INT =
+      new byte[] {VariantTestUtil.primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49};
   private static final byte[] VALUE_STRING =
-      new byte[] {primitiveHeader(16), 0x07, 0x00, 0x00, 0x00, 'v', 'a', 'r', 'i', 'a', 'n', 't'};
+      new byte[] {VariantTestUtil.primitiveHeader(16), 0x07, 0x00, 0x00, 0x00, 'v', 'a', 'r', 'i', 'a', 'n', 't'};

Review Comment:
   Split out the builder tests to their own files.



##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantObjectBuilder.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant object, used by VariantBuilder.
+ */
+public class VariantObjectBuilder {

Review Comment:
   Yeah, I updated `VariantObjectBuilder` so it validates the alternating of `appendKey` and `appendValue`, and added incorrect api usage tests.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObject.java:
##########
@@ -250,22 +212,61 @@ public void testMixedObject() {
     Variant value = new Variant(
         ByteBuffer.wrap(constructObject(keys, fields, true)),
         constructMetadata(true, ImmutableList.of("a", "b", "c")));
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
       Assert.assertEquals(3, v.numObjectElements());
-      checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
+      VariantTestUtil.checkType(v.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.INT);
       Assert.assertEquals(1234567890, v.getFieldByKey("a").getInt());
-      checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
+      VariantTestUtil.checkType(v.getFieldByKey("b"), VariantUtil.PRIMITIVE, Variant.Type.BOOLEAN);
       Assert.assertTrue(v.getFieldByKey("b").getBoolean());
-      checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
+      VariantTestUtil.checkType(v.getFieldByKey("c"), VariantUtil.OBJECT, Variant.Type.OBJECT);
 
       Variant nestedV = v.getFieldByKey("c");
       Assert.assertEquals(2, nestedV.numObjectElements());
-      checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("a"), VariantUtil.PRIMITIVE, Variant.Type.DATE);
       Assert.assertEquals(
           LocalDate.parse("2025-04-17"),
           LocalDate.ofEpochDay(nestedV.getFieldByKey("a").getInt()));
-      checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+      VariantTestUtil.checkType(nestedV.getFieldByKey("c"), VariantUtil.PRIMITIVE, Variant.Type.NULL);
+    });
+  }
+
+  @Test
+  public void testMixedObjectBuilder() {
+    VariantBuilder b = new VariantBuilder(true);
+    VariantObjectBuilder objBuilder = b.startObject();
+    b.appendObjectKey(objBuilder, "outer 1");
+    b.appendBoolean(true);
+    b.appendObjectKey(objBuilder, "outer 2");
+    b.appendLong(1234567890);
+    b.appendObjectKey(objBuilder, "outer 3");
+    {
+      // build a nested obj
+      VariantObjectBuilder nestedBuilder = b.startObject();
+      b.appendObjectKey(nestedBuilder, "nested 1");
+      {
+        // build a nested empty obj
+        VariantObjectBuilder emptyBuilder = b.startObject();
+        b.endObject(emptyBuilder);
+      }
+      b.appendObjectKey(nestedBuilder, "nested 2");
+      b.appendString("variant");
+      b.endObject(nestedBuilder);
+    }
+    b.endObject(objBuilder);
+
+    VariantTestUtil.testVariant(b.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.OBJECT, Variant.Type.OBJECT);
+      Assert.assertEquals(3, v.numObjectElements());
+      Assert.assertTrue(v.getFieldByKey("outer 1").getBoolean());
+      Assert.assertEquals(1234567890, v.getFieldByKey("outer 2").getLong());

Review Comment:
   Added type checks.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -451,52 +630,97 @@ public void testNegativeTimestampNtz() {
           (byte) 0xFF,
           (byte) 0xFF
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_NTZ);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_NTZ);
       Assert.assertEquals(
           Instant.parse("1969-12-31T23:59:59.999999Z"), Instant.EPOCH.plus(v.getLong(), ChronoUnit.MICROS));
     });
   }
 
+  @Test
+  public void testTimestampNtzBuilder() {
+    DateTimeFormatter dtf = DateTimeFormatter.ISO_DATE_TIME;
+    VariantBuilder vb = new VariantBuilder(false);
+    long micros = VariantTestUtil.microsSinceEpoch(Instant.from(dtf.parse("2024-01-01T23:00:00.000001Z")));
+    vb.appendTimestampNtz(micros);
+    VariantTestUtil.testVariant(vb.build(), v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIMESTAMP_NTZ);
+      Assert.assertEquals(micros, v.getLong());
+    });
+  }
+
   @Test
   public void testBinary() {
     Variant value = new Variant(
-        ByteBuffer.wrap(new byte[] {primitiveHeader(15), 0x05, 0x00, 0x00, 0x00, 'a', 'b', 'c', 'd', 'e'}),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BINARY);
+        ByteBuffer.wrap(
+            new byte[] {VariantTestUtil.primitiveHeader(15), 0x05, 0x00, 0x00, 0x00, 'a', 'b', 'c', 'd', 'e'
+            }),
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.BINARY);
       Assert.assertEquals(ByteBuffer.wrap(new byte[] {'a', 'b', 'c', 'd', 'e'}), v.getBinary());
     });
   }
 
+  @Test
+  public void testBinaryBuilder() {
+    VariantBuilder vb = new VariantBuilder(false);
+    byte[] binary = new byte[] {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};
+    vb.appendBinary(binary);

Review Comment:
   Updated.



##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantScalar.java:
##########
@@ -506,32 +730,54 @@ public void testTime() {
           (byte) 0x00,
           (byte) 0x00
         }),
-        EMPTY_METADATA);
-    testVariant(value, v -> {
-      checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIME);
+        VariantTestUtil.EMPTY_METADATA);
+    VariantTestUtil.testVariant(value, v -> {
+      VariantTestUtil.checkType(v, VariantUtil.PRIMITIVE, Variant.Type.TIME);
       Assert.assertEquals(LocalTime.parse("23:59:59.123456"), LocalTime.ofNanoOfDay(v.getLong() * 1_000));
     });
   }
 
   @Test
-  public void testTimestampNanos() {
+  public void testTimeBuilder() {
+    for (String timeStr : Arrays.asList(
+        "00:00:00.000000", "00:00:00.000120", "12:00:00.000000", "12:00:00.002300", "23:59:59.999999")) {
+      VariantBuilder vb = new VariantBuilder(false);
+      long micros = LocalTime.parse(timeStr).toNanoOfDay() / 1_000;
+      vb.appendTime(micros);

Review Comment:
   I added a check to disallow negative values.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2069555833


##########
parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantArray.java:
##########
@@ -20,104 +20,32 @@
 
 import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
-import java.nio.charset.StandardCharsets;
-import java.security.SecureRandom;
 import java.time.LocalDate;
-import java.util.*;
-import java.util.function.Consumer;
 import org.junit.Assert;
 import org.junit.Test;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 public class TestVariantArray {
   private static final Logger LOG = LoggerFactory.getLogger(TestVariantArray.class);
-  private static final String RANDOM_CHARS = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
 
-  /** Random number generator for generating random strings */
-  private static SecureRandom random = new SecureRandom(new byte[] {1, 2, 3, 4, 5});
-
-  private static final ByteBuffer EMPTY_METADATA = ByteBuffer.wrap(new byte[] {0b1});
-
-  private static final byte[] VALUE_NULL = new byte[] {primitiveHeader(0)};
-  private static final byte[] VALUE_BOOL = new byte[] {primitiveHeader(1)};
-  private static final byte[] VALUE_INT = new byte[] {primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49};
+  private static final byte[] VALUE_NULL = new byte[] {VariantTestUtil.primitiveHeader(0)};
+  private static final byte[] VALUE_BOOL = new byte[] {VariantTestUtil.primitiveHeader(1)};
+  private static final byte[] VALUE_INT =
+      new byte[] {VariantTestUtil.primitiveHeader(5), (byte) 0xD2, 0x02, (byte) 0x96, 0x49};
   private static final byte[] VALUE_STRING =
-      new byte[] {primitiveHeader(16), 0x07, 0x00, 0x00, 0x00, 'v', 'a', 'r', 'i', 'a', 'n', 't'};
+      new byte[] {VariantTestUtil.primitiveHeader(16), 0x07, 0x00, 0x00, 0x00, 'v', 'a', 'r', 'i', 'a', 'n', 't'};

Review Comment:
   Overall, I think the tests are covering relevant encoding cases. I'd prefer encoding and decoding tests in separate suites to be able to maintain them better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org

Re: [PR] GH-3201: Implement a Variant builder to create Variant values [parquet-java]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #3202:
URL: https://github.com/apache/parquet-java/pull/3202#discussion_r2072156852


##########
parquet-variant/src/main/java/org/apache/parquet/variant/VariantObjectBuilder.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.parquet.variant;
+
+import java.util.ArrayList;
+
+/**
+ * Builder for creating Variant object, used by VariantBuilder.
+ */
+public class VariantObjectBuilder extends VariantBuilder {
+  /** The parent VariantBuilder. */
+  private final VariantBuilder parent;
+  /** The FieldEntry list for the fields of this object. */
+  private final ArrayList<VariantBuilder.FieldEntry> fields;
+  /** The number of values appended to this object. */
+  protected long numValues = 0;
+
+  VariantObjectBuilder(VariantBuilder parent) {
+    this.parent = parent;
+    this.fields = new ArrayList<>();
+  }
+
+  /**
+   * Appends an object key to this object. This method must be called before appending any value.
+   * @param key the key to append
+   */
+  void appendKey(String key) {
+    if (fields.size() > numValues) {
+      throw new IllegalStateException("Cannot call appendKey() before appending an object value.");

Review Comment:
   I think this is technically "before appending a value for the previous key" right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org