[PR] Parquet: Internal writer and reader [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat opened a new pull request, #11904:
URL: https://github.com/apache/iceberg/pull/11904

   Splitted into 3 commits, 
   
   a) Refactor BaseParquetWriter to only keep common functionality required for internal and generic writer.
   b) Refactor BaseParquetReaders to only keep common functionality required for internal and generic reader.
   c) Add internal writer and reader that consumes and produces the Iceberg in-memory data model.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917319575


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new StructLikeReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampInt96Reader(desc);
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return Optional.of(new ParquetValueReaders.TimestampMillisReader(desc));
+    }
+
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit, boolean isAdjustedToUTC) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return Optional.of(new ParquetValueReaders.TimestampMillisReader(desc));
+    }
+
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {
+    return Optional.of(new ParquetValueReaders.UUIDReader(desc));
+  }
+
+  private static class StructLikeReader extends StructReader<StructLike, StructLike> {
+    private final GenericRecord template;
+
+    StructLikeReader(List<Type> types, List<ParquetValueReader<?>> readers, StructType struct) {
+      super(types, readers);

Review Comment:
   The superclass doesn't use `types` so it doesn't need to be passed in. This isn't a big deal if you don't want to fix the superclass in this PR though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906095465


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);

Review Comment:
   This isn't correct. The unboxed reader will return a `Binary` for int96 columns. Instead, this needs to use the same logic as the Spark reader (which also uses the internal representation):
   
   ```java
     private static class TimestampInt96Reader extends UnboxedReader<Long> {
       TimestampInt96Reader(ColumnDescriptor desc) {
         super(desc);
       }
   
       @Override
       public Long read(Long ignored) {
         return readLong();
       }
       @Override
       public long readLong() {
         final ByteBuffer byteBuffer =
             column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
         return ParquetUtil.extractTimestampInt96(byteBuffer);
       }
     }
   ```
   
   You can move that class into the `parquet` package to share it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1920247487


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +42,27 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return ParquetValueWriters.fixed(desc);
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> dateWriter(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> timeWriter(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> uuidWriter(ColumnDescriptor desc) {

Review Comment:
   CI just failed for ArrowReaderTest > testReadAll().
   
   Generic writer expects byte[] based UUID handling and internal reader and writer works directly on UUID object. So, we do need a factory method here as we need to supply different kinds of UUID representation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925898084


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -850,4 +919,42 @@ private TripleIterator<?> firstNonNullColumn(List<TripleIterator<?>> columns) {
       return NullReader.NULL_COLUMN;
     }
   }
+
+  private static class RecordReader<T extends StructLike> extends StructReader<T, T> {
+    private final GenericRecord template;
+
+    private RecordReader(
+        List<Type> types, List<ParquetValueReader<?>> readers, Types.StructType struct) {
+      super(types, readers);
+      this.template = struct != null ? GenericRecord.create(struct) : null;
+    }
+
+    @Override
+    protected T newStructData(T reuse) {
+      if (reuse != null) {
+        return reuse;
+      } else {
+        // GenericRecord.copy() is more performant then GenericRecord.create(StructType) since
+        // NAME_MAP_CACHE access
+        // is eliminated. Using copy here to gain performance.

Review Comment:
   Can you fix the formatting of this comment since it has moved?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1919615649


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -397,7 +404,7 @@ public ParquetValueReader<?> primitive(
         case INT96:
           // Impala & Spark used to write timestamps as INT96 without a logical type. For backwards
           // compatibility we try to read INT96 as timestamps.
-          return new TimestampInt96Reader(desc);
+          return int96Reader(desc);

Review Comment:
   I should assume `isAdjustedToUTC` to be true? because primitive don't have `TimestampType` and the reader was using `OffsetDateTime`, 
   
   my understanding is if  `OffsetDateTime` is used it is with timezone and `LocalDateTime` is without timezone. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918344913


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -92,4 +151,124 @@ protected void set(Record struct, int pos, Object value) {
       struct.set(pos, value);
     }
   }
+
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  private static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {
+    private DateReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDate read(LocalDate reuse) {
+      return EPOCH_DAY.plusDays(column.nextInteger());
+    }
+  }
+
+  private static class TimestampReader extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampMillisReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampInt96Reader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private static final long UNIX_EPOCH_JULIAN = 2_440_588L;
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      final long timeOfDayNanos = byteBuffer.getLong();
+      final int julianDay = byteBuffer.getInt();
+
+      return Instant.ofEpochMilli(TimeUnit.DAYS.toMillis(julianDay - UNIX_EPOCH_JULIAN))
+          .plusNanos(timeOfDayNanos)
+          .atOffset(ZoneOffset.UTC);
+    }
+  }
+
+  private static class TimestamptzReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimestamptzMillisReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimeMillisReader extends ParquetValueReaders.PrimitiveReader<LocalTime> {
+    private TimeMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalTime read(LocalTime reuse) {
+      return LocalTime.ofNanoOfDay(column.nextLong() * 1000000L);

Review Comment:
   how do I test these? `DataTest` doesn't handle time units and writers write in micro always? 
   Do you mean existing code is wrong? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917278880


##########
data/src/test/java/org/apache/iceberg/data/parquet/TestInternalData.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.InternalTestHelpers;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RandomInternalData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.DataTest;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+public class TestInternalData extends DataTest {

Review Comment:
   The Avro equivalent is named `TestInternalAvro`. Why not `TestInternalParquet` instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917518515


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   I didn't do it because, I can't move the date/time classes to `GenericReader`. It has to be in this class. But yeah, I got your point about behavior change.  I will revert that date/time classes movement. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924339986


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -192,13 +222,17 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      return Optional.ofNullable(dateWriter(desc));
     }
 
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeType) {
-      return Optional.of(new TimeWriter(desc));
+      Preconditions.checkArgument(
+          LogicalTypeAnnotation.TimeUnit.MICROS.equals(timeType.getUnit()),
+          "Cannot write time in %s, only MICROS is supported",
+          timeType.getUnit());
+      return Optional.ofNullable(timeWriter(desc));

Review Comment:
   Why does this use `ofNullable`? The intent is not to expose `BaseParquetWriter` beyond its package, so there is no need to handle cases where the factory method return `null`. Those are error cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925899152


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -850,4 +919,42 @@ private TripleIterator<?> firstNonNullColumn(List<TripleIterator<?>> columns) {
       return NullReader.NULL_COLUMN;
     }
   }
+
+  private static class RecordReader<T extends StructLike> extends StructReader<T, T> {

Review Comment:
   This returns `Record` so I don't think it needed to be modified. It doesn't return any other subclass of `StructLike`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906068027


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Optional;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> buildWriter(MessageType type) {
+    return INSTANCE.createWriter(type);
+  }
+
+  @Override
+  protected StructWriter<StructLike> createStructWriter(List<ParquetValueWriter<?>> writers) {
+    return new ParquetStructWriter(writers);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<
+          ParquetValueWriters.PrimitiveWriter<?>>
+      logicalTypeWriterVisitor(ColumnDescriptor desc) {
+    return new LogicalTypeWriterVisitor(desc);
+  }
+
+  @Override
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    // accepts ByteBuffer and internally writes as binary.
+    return ParquetValueWriters.byteBuffers(desc);

Review Comment:
   Make sure this writer checks the length of the incoming bytes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906132860


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -92,4 +127,232 @@ protected void set(Record struct, int pos, Object value) {
       struct.set(pos, value);
     }
   }
+
+  private class LogicalTypeAnnotationParquetValueReaderVisitor
+      implements LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>> {
+
+    private final ColumnDescriptor desc;
+    private final org.apache.iceberg.types.Type.PrimitiveType expected;
+    private final PrimitiveType primitive;
+
+    LogicalTypeAnnotationParquetValueReaderVisitor(
+        ColumnDescriptor desc,
+        org.apache.iceberg.types.Type.PrimitiveType expected,
+        PrimitiveType primitive) {
+      this.desc = desc;
+      this.expected = expected;
+      this.primitive = primitive;
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.StringLogicalTypeAnnotation stringLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.EnumLogicalTypeAnnotation enumLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalLogicalType) {
+      switch (primitive.getPrimitiveTypeName()) {
+        case BINARY:
+        case FIXED_LEN_BYTE_ARRAY:
+          return Optional.of(
+              new ParquetValueReaders.BinaryAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT64:
+          return Optional.of(
+              new ParquetValueReaders.LongAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT32:
+          return Optional.of(
+              new ParquetValueReaders.IntegerAsDecimalReader(desc, decimalLogicalType.getScale()));
+        default:
+          throw new UnsupportedOperationException(
+              "Unsupported base type for decimal: " + primitive.getPrimitiveTypeName());
+      }
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DateLogicalTypeAnnotation dateLogicalType) {
+      return Optional.of(new DateReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeLogicalType) {
+      if (timeLogicalType.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS) {
+        return Optional.of(new TimeReader(desc));
+      } else if (timeLogicalType.getUnit() == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+        return Optional.of(new TimeMillisReader(desc));
+      }
+
+      return Optional.empty();
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestampLogicalType) {
+      if (timestampLogicalType.getUnit() == LogicalTypeAnnotation.TimeUnit.MICROS) {
+        Types.TimestampType tsMicrosType = (Types.TimestampType) expected;
+        return tsMicrosType.shouldAdjustToUTC()
+            ? Optional.of(new TimestamptzReader(desc))
+            : Optional.of(new TimestampReader(desc));
+      } else if (timestampLogicalType.getUnit() == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+        Types.TimestampType tsMillisType = (Types.TimestampType) expected;
+        return tsMillisType.shouldAdjustToUTC()
+            ? Optional.of(new TimestamptzMillisReader(desc))
+            : Optional.of(new TimestampMillisReader(desc));
+      }
+
+      return LogicalTypeAnnotation.LogicalTypeAnnotationVisitor.super.visit(timestampLogicalType);
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.IntLogicalTypeAnnotation intLogicalType) {
+      if (intLogicalType.getBitWidth() == 64) {
+        return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+      }
+      return (expected.typeId() == org.apache.iceberg.types.Type.TypeID.LONG)
+          ? Optional.of(new ParquetValueReaders.IntAsLongReader(desc))
+          : Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.JsonLogicalTypeAnnotation jsonLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.BsonLogicalTypeAnnotation bsonLogicalType) {
+      return Optional.of(new ParquetValueReaders.BytesReader(desc));
+    }
+  }
+
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  private static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {

Review Comment:
   I agree with moving the date/time reader classes here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1908235843


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Optional;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> buildWriter(MessageType type) {
+    return INSTANCE.createWriter(type);
+  }
+
+  @Override
+  protected StructWriter<StructLike> createStructWriter(List<ParquetValueWriter<?>> writers) {
+    return new ParquetStructWriter(writers);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<
+          ParquetValueWriters.PrimitiveWriter<?>>
+      logicalTypeWriterVisitor(ColumnDescriptor desc) {
+    return new LogicalTypeWriterVisitor(desc);
+  }
+
+  @Override
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    // accepts ByteBuffer and internally writes as binary.
+    return ParquetValueWriters.byteBuffers(desc);

Review Comment:
   ok. Also, the existing code of `GenericParquetWriter` also not having this length check while writing as byte[], I will add that check.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927639914


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {

Review Comment:
   Minor: The type this returns should be `StructLike` rather than `Record`. While this will return `Record`, there is no need to require it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906138972


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {
+    private final GenericRecord template;
+
+    ParquetStructReader(List<Type> types, List<ParquetValueReader<?>> readers, StructType struct) {
+      super(types, readers);
+      this.template = struct != null ? GenericRecord.create(struct) : null;
+    }
+
+    @Override
+    protected StructLike newStructData(StructLike reuse) {
+      if (reuse != null) {
+        return reuse;
+      } else {
+        return template.copy();
+      }
+    }
+
+    @Override
+    protected Object getField(StructLike intermediate, int pos) {
+      return intermediate.get(pos, Object.class);
+    }
+
+    @Override
+    protected StructLike buildStruct(StructLike struct) {
+      return struct;
+    }
+
+    @Override
+    protected void set(StructLike struct, int pos, Object value) {
+      struct.set(pos, value);
+    }
+  }
+
+  private static class LogicalTypeAnnotationParquetValueReaderVisitor
+      implements LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>> {
+
+    private final ColumnDescriptor desc;
+    private final org.apache.iceberg.types.Type.PrimitiveType expected;
+    private final PrimitiveType primitive;
+
+    LogicalTypeAnnotationParquetValueReaderVisitor(
+        ColumnDescriptor desc,
+        org.apache.iceberg.types.Type.PrimitiveType expected,
+        PrimitiveType primitive) {
+      this.desc = desc;
+      this.expected = expected;
+      this.primitive = primitive;
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.StringLogicalTypeAnnotation stringLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.EnumLogicalTypeAnnotation enumLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.UUIDLogicalTypeAnnotation uuidLogicalType) {
+      return Optional.of(new ParquetValueReaders.UUIDReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalLogicalType) {
+      switch (primitive.getPrimitiveTypeName()) {
+        case BINARY:
+        case FIXED_LEN_BYTE_ARRAY:
+          return Optional.of(
+              new ParquetValueReaders.BinaryAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT64:
+          return Optional.of(
+              new ParquetValueReaders.LongAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT32:
+          return Optional.of(
+              new ParquetValueReaders.IntegerAsDecimalReader(desc, decimalLogicalType.getScale()));
+        default:
+          throw new UnsupportedOperationException(
+              "Unsupported base type for decimal: " + primitive.getPrimitiveTypeName());
+      }
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DateLogicalTypeAnnotation dateLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));

Review Comment:
   This isn't correct. Like timestamp, this needs to handle the unit of the incoming value. In addition, millisecond values must annotate an int32 according to the [Parquet logical type docs](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#time). When the unit is a millisecond value, this needs to call `readInt` and multiply by 1000.
   
   It looks like both Spark currently gets the underlying Parquet type for milliseconds wrong (which makes sense because this is never used in Spark). We can go ahead and fix this now and share the reader between Internal and Spark.
   
   ```java
     private static class TimestampMillisReader extends UnboxedReader<Long> {
       TimestampMillisReader(ColumnDescriptor desc) {
         super(desc);
       }
   
       @Override
       public Long read(Long ignored) {
         return readLong();
       }
   
       @Override
       public long readLong() {
         return 1000L * column.nextInteger();
       }
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1911927458


##########
api/src/test/java/org/apache/iceberg/util/RandomUtil.java:
##########
@@ -237,7 +237,7 @@ private static BigInteger randomUnscaled(int precision, Random random) {
   }
 
   public static List<Object> generateList(
-      Random random, Types.ListType list, Supplier<Object> elementResult) {

Review Comment:
   Addressing the nit from https://github.com/apache/iceberg/pull/11919/#discussion_r1909246082



##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -192,13 +205,17 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      return dateWriter(desc);
     }
 
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeType) {
-      return Optional.of(new TimeWriter(desc));
+      Preconditions.checkArgument(

Review Comment:
   Added this check as for timestamp there was a check to process only MICROS. 



##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueWriters.java:
##########
@@ -313,6 +327,44 @@ public void write(int repetitionLevel, ByteBuffer buffer) {
     }
   }
 
+  private static class FixedBufferWriter extends PrimitiveWriter<ByteBuffer> {
+    private final int length;
+
+    private FixedBufferWriter(ColumnDescriptor desc) {
+      super(desc);
+      this.length = desc.getPrimitiveType().getTypeLength();
+    }
+
+    @Override
+    public void write(int repetitionLevel, ByteBuffer buffer) {
+      Preconditions.checkArgument(
+          buffer.remaining() == length,
+          "Cannot write byte buffer of length %s as fixed[%s]",
+          buffer.remaining(),
+          length);
+      column.writeBinary(repetitionLevel, Binary.fromReusedByteBuffer(buffer));
+    }
+  }
+
+  private static class FixedWriter extends PrimitiveWriter<byte[]> {

Review Comment:
   I just moved this class from `BaseParquetWriter`, it was missing validation for length. I added it.  



##########
data/src/test/java/org/apache/iceberg/data/parquet/TestInternalData.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.InternalTestHelpers;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RandomInternalData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.DataTest;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+public class TestInternalData extends DataTest {

Review Comment:
   Kept this class in data module instead of parquet because, parquet module don't have `DataTest`. It will be lot of code duplication to have it in parquet module. 



##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -208,11 +225,7 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
           LogicalTypeAnnotation.TimeUnit.MICROS.equals(timestampType.getUnit()),
           "Cannot write timestamp in %s, only MICROS is supported",
           timestampType.getUnit());
-      if (timestampType.isAdjustedToUTC()) {
-        return Optional.of(new TimestamptzWriter(desc));
-      } else {
-        return Optional.of(new TimestampWriter(desc));
-      }
+      return timestampWriter(desc, timestampType.isAdjustedToUTC());

Review Comment:
   Should I pass a `TimeUnit` to avoid modifying the signature in the future when other units are supported?  Also same for timeType



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetReaders.java:
##########
@@ -373,41 +371,6 @@ public Decimal read(Decimal ignored) {
     }
   }
 
-  private static class TimestampMillisReader extends UnboxedReader<Long> {

Review Comment:
   Only removed for latest version of spark, thinking when we deprecate the older versions, this code will be removed. 
   Spark-3.3 is deprecated. Spark-3.4 Should I handle it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1928003465


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -412,6 +446,41 @@ public byte[] read(byte[] ignored) {
     }
   }
 
+  private static class TimestampInt96Reader extends UnboxedReader<Long> {
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      return ParquetUtil.extractTimestampInt96(byteBuffer);
+    }
+  }
+
+  private static class TimestampMillisReader extends UnboxedReader<Long> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      return 1000L * column.nextInteger();

Review Comment:
   Lack of test coverage in the base code for these milliseconds time, timestamp and int96 timestamps is the reason for back and forth. Tests would have caught this. Will try to add in a follow up.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924336176


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +80,46 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReader<?> fixedReader(ColumnDescriptor desc) {
+    return new FixedReader(desc);
+  }
+
+  protected ParquetValueReader<?> dateReader(ColumnDescriptor desc) {
+    return new DateReader(desc);
+  }
+
+  protected ParquetValueReader<?> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    switch (unit) {
+      case MICROS:
+        return new TimeReader(desc);
+      case MILLIS:
+        return new TimeMillisReader(desc);
+      default:
+        throw new UnsupportedOperationException("Unsupported Unit: " + unit);
+    }
+  }
+
+  protected ParquetValueReader<?> timestampReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit, boolean isAdjustedToUTC) {
+    switch (unit) {
+      case MICROS:
+        return isAdjustedToUTC ? new TimestamptzReader(desc) : new TimestampReader(desc);
+      case MILLIS:
+        return isAdjustedToUTC
+            ? new TimestamptzMillisReader(desc)
+            : new TimestampMillisReader(desc);
+      case NANOS:
+        if (isAdjustedToUTC) {
+          return new TimestampInt96Reader(desc);

Review Comment:
   This isn't correct. The INT96 timestamp reader is used when the underlying type is INT96, not when the Iceberg or Parquet type has nanosecond precision. Nanosecond precision timestamps were recently introduced and are not used to signal that Parquet stores timestamp as an INT96.
   
   This should match the previous behavior, where this is used when underlying physical type (`desc.getPrimitiveType().getPrimitiveTypeName()`) is INT96.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925911079


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -63,6 +69,23 @@ public static ParquetValueReader<Long> position() {
     return new PositionReader();
   }
 
+  public static ParquetValueReader<UUID> uuids(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UUIDReader(desc);
+  }
+
+  public static ParquetValueReader<Long> int96Timestamps(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampInt96Reader(desc);
+  }
+
+  public static ParquetValueReader<Long> millisAsTimestamps(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampMillisReader(desc);
+  }
+
+  public static <T extends StructLike> StructReader<T, T> recordReader(

Review Comment:
   Should be `ParquetValueReader<Record>`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#issuecomment-2609690162

   @rdblue: Thanks for giving additional context for unresolved comments. I think I understood all the comments this time. PR is ready. It also fixed base code issues and testcases. 
   
   
   
   
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924386754


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   @ajantha-bhat, why can't the reader classes be moved if this returns an instance? The classes are currently `private` so there is no direct dependency on them. As long as an instance is returned, wouldn't it work?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924387728


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {

Review Comment:
   These methods should return `ParquetValueWriter<?>` and not a specific class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925901471


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +70,46 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReader<?> fixedReader(ColumnDescriptor desc) {
+    return new GenericParquetReaders.FixedReader(desc);
+  }
+
+  protected ParquetValueReader<?> dateReader(ColumnDescriptor desc) {
+    return new GenericParquetReaders.DateReader(desc);
+  }
+
+  protected ParquetValueReader<?> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    switch (unit) {
+      case MICROS:
+        return new GenericParquetReaders.TimeReader(desc);
+      case MILLIS:
+        return new GenericParquetReaders.TimeMillisReader(desc);
+      default:
+        throw new UnsupportedOperationException("Unsupported Unit: " + unit);

Review Comment:
   Error messages should use sentence case, so this should be `"Unsupported unit for time: "`. It is also nice to give additional context in the message (the "for time" part).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917311413


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(

Review Comment:
   Most object models use `create` instead of `buildReader`. I think it would make sense to match that convention rather than copy from `GenericParquetReaders`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925683533


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +80,46 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReader<?> fixedReader(ColumnDescriptor desc) {
+    return new FixedReader(desc);
+  }
+
+  protected ParquetValueReader<?> dateReader(ColumnDescriptor desc) {
+    return new DateReader(desc);
+  }
+
+  protected ParquetValueReader<?> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    switch (unit) {
+      case MICROS:
+        return new TimeReader(desc);
+      case MILLIS:
+        return new TimeMillisReader(desc);
+      default:
+        throw new UnsupportedOperationException("Unsupported Unit: " + unit);
+    }
+  }
+
+  protected ParquetValueReader<?> timestampReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit, boolean isAdjustedToUTC) {
+    switch (unit) {
+      case MICROS:
+        return isAdjustedToUTC ? new TimestamptzReader(desc) : new TimestampReader(desc);
+      case MILLIS:
+        return isAdjustedToUTC
+            ? new TimestamptzMillisReader(desc)
+            : new TimestampMillisReader(desc);
+      case NANOS:
+        if (isAdjustedToUTC) {
+          return new TimestampInt96Reader(desc);

Review Comment:
   I see. In that case I feel maybe previous changes of separate factory method before https://github.com/apache/iceberg/pull/11904#discussion_r1917285311 was more suitable. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924343948


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -63,6 +69,18 @@ public static ParquetValueReader<Long> position() {
     return new PositionReader();
   }
 
+  public static ParquetValueReader<UUID> uuids(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UUIDReader(desc);
+  }
+
+  public static ParquetValueReader<Long> timestampInt96Reader(ColumnDescriptor desc) {

Review Comment:
   To match the other method names, I think this should be `int96Timestamps`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925923531


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new DateWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new TimeWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new TimestamptzWriter(desc);
+    } else {
+      return new TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer; no special writer needed.

Review Comment:
   That doesn't work because the UUID representation is `UUID`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927639914


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {

Review Comment:
   Minor: The type this returns should be `StructLike` rather than `Record`. While this will return `Record`, there is no need to require it. It is also inaccurate because the internal readers will be able to instantiate other `StructLike` records for some types to match the behavior in Avro eventually.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1928001378


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {

Review Comment:
   Thanks. I merged the PR to my branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#issuecomment-2585142658

   @rdblue: Thanks for the review. I have addressed the comments. Please take a look at it again.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927632018


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetReaders.java:
##########
@@ -265,7 +263,7 @@ public ParquetValueReader<?> primitive(
           case TIMESTAMP_MICROS:
             return new UnboxedReader<>(desc);
           case TIMESTAMP_MILLIS:
-            return new TimestampMillisReader(desc);
+            return ParquetValueReaders.millisAsTimestamps(desc);

Review Comment:
   This should also be `millisAsTimes`



##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetReaders.java:
##########
@@ -265,7 +263,7 @@ public ParquetValueReader<?> primitive(
           case TIMESTAMP_MICROS:
             return new UnboxedReader<>(desc);
           case TIMESTAMP_MILLIS:
-            return new TimestampMillisReader(desc);
+            return ParquetValueReaders.millisAsTimestamps(desc);

Review Comment:
   This should also be `millisAsTimes`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917317907


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new StructLikeReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampInt96Reader(desc);
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return Optional.of(new ParquetValueReaders.TimestampMillisReader(desc));
+    }
+
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit, boolean isAdjustedToUTC) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return Optional.of(new ParquetValueReaders.TimestampMillisReader(desc));
+    }
+
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {
+    return Optional.of(new ParquetValueReaders.UUIDReader(desc));
+  }
+
+  private static class StructLikeReader extends StructReader<StructLike, StructLike> {

Review Comment:
   Now I'm questioning the use of `StructLikeReader` vs `RecordReader` in `GenericParquetReaders`. They both produce the same implementation class (`GenericRecord`), and `Record` is a `StructLike`. Why not move `RecordReader` into `ParquetValueReaders` and reuse the code?
   
   That would also preserve the comment in `RecordReader` that was removed when copying here, but is still relevant to why there is a `template` record.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1919619088


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timeReader(ColumnDescriptor desc, TimeUnit unit) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, TimeUnit unit, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {

Review Comment:
   I see. So the Generic writers and readers never handled UUID !



##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timeReader(ColumnDescriptor desc, TimeUnit unit) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, TimeUnit unit, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {

Review Comment:
   CI just failed for `ArrowReaderTest > testReadAll()`. 
   
   Generic reader and writer expects byte[] based UUID handling and internal reader and writer works directly on UUID object. So, we do need a factory method here as we need to supply different kinds of UUID representation. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918335337


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -397,7 +404,7 @@ public ParquetValueReader<?> primitive(
         case INT96:
           // Impala & Spark used to write timestamps as INT96 without a logical type. For backwards
           // compatibility we try to read INT96 as timestamps.
-          return new TimestampInt96Reader(desc);
+          return int96Reader(desc);

Review Comment:
   We don't have a nano handling in `timestampReader` yet. So, you are suggesting to rename `TimestampInt96Reader` to `TimestampNanoReader` and call it from `timestampReader`? So, it can also be used for nano reading in future? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906077444


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {

Review Comment:
   When checking that name (`RecordReader`) for consistency, I noticed that there's already a `RecordReader` in `GenericParquetReaders`. You can reuse that class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927629344


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -412,6 +446,41 @@ public byte[] read(byte[] ignored) {
     }
   }
 
+  private static class TimestampInt96Reader extends UnboxedReader<Long> {
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      return ParquetUtil.extractTimestampInt96(byteBuffer);
+    }
+  }
+
+  private static class TimestampMillisReader extends UnboxedReader<Long> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      return 1000L * column.nextInteger();

Review Comment:
   This is valid for `time` but not for `timestamp`. I may have mixed up the `timestamp` reader and `time` reader in an earlier comment. This needs to be `nextLong`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924346731


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -63,6 +69,18 @@ public static ParquetValueReader<Long> position() {
     return new PositionReader();
   }
 
+  public static ParquetValueReader<UUID> uuids(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UUIDReader(desc);
+  }
+
+  public static ParquetValueReader<Long> timestampInt96Reader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampInt96Reader(desc);
+  }
+
+  public static ParquetValueReader<Long> timestampMillisReader(ColumnDescriptor desc) {

Review Comment:
   I'd probably rename this as well, maybe `millisAsTimestamps` or `millisecondTimestamps`



##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -63,6 +69,18 @@ public static ParquetValueReader<Long> position() {
     return new PositionReader();
   }
 
+  public static ParquetValueReader<UUID> uuids(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UUIDReader(desc);
+  }
+
+  public static ParquetValueReader<Long> timestampInt96Reader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampInt96Reader(desc);
+  }
+
+  public static ParquetValueReader<Long> timestampMillisReader(ColumnDescriptor desc) {

Review Comment:
   I'd probably rename this as well, maybe `millisAsTimestamps`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927639914


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {

Review Comment:
   Minor: The type this returns should use `StructLike` rather than `Record`. While this will return `Record`, there is no need to require it. And like the Avro readers, we plan to extend this to be able to return custom types for specific structures. That way we can use it to produce things like `ManifestFile`.
   
   I was able to get the types working like this:
   ```java
   public class InternalReader<T extends StructLike> extends BaseParquetReaders<T> {
   
     private static final InternalReader<?> INSTANCE = new InternalReader<>();
   
     private InternalReader() {}
   
     @SuppressWarnings("unchecked")
     public static <T extends StructLike> ParquetValueReader<T> create(Schema expectedSchema, MessageType fileSchema) {
       return (ParquetValueReader<T>) INSTANCE.createReader(expectedSchema, fileSchema);
     }
   
     @SuppressWarnings("unchecked")
     public static <T extends StructLike> ParquetValueReader<T> create(
         Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
       return (ParquetValueReader<T>) INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
     }
   
     @Override
     @SuppressWarnings("unchecked")
     protected ParquetValueReader<T> createStructReader(
         List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
       return (ParquetValueReader<T>) ParquetValueReaders.recordReader(types, fieldReaders, structType);
     }
   
     ...
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917303018


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {

Review Comment:
   These should not return `Optional`:
   * Iceberg generally prefers returning `null` over using `Optional`
   * The existing methods (like `createStructReader`) do not wrap return types in `Optional`
   * These methods should always return a valid reader, or should throw an exception to signal that something is not supported



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917500263


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +42,27 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return ParquetValueWriters.fixed(desc);
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> dateWriter(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> timeWriter(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueWriters.PrimitiveWriter<?>> uuidWriter(ColumnDescriptor desc) {

Review Comment:
   Feedback here is similar to the read path:
   * Should not use Optional
   * Should produce the same writer as before
   * Should deprecate this class so that we can make it package-private
   * Should not need a factory method for UUID



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925912623


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -190,33 +210,49 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     }
 
     @Override
-    public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
+    public Optional<ParquetValueWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      ParquetValueWriter<?> dateWriter = dateWriter(desc);
+      if (dateWriter != null) {
+        return Optional.of(dateWriter(desc));
+      }
+
+      return LogicalTypeAnnotation.LogicalTypeAnnotationVisitor.super.visit(dateType);

Review Comment:
   Is your intent to not support dates, times, or timestamps for the internal model? If so, why?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925927030


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -49,47 +59,132 @@ public static ParquetValueReader<Record> buildReader(
   @Override
   protected ParquetValueReader<Record> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
-    return new RecordReader(types, fieldReaders, structType);
+    return ParquetValueReaders.recordReader(types, fieldReaders, structType);
   }
 
   @Override
   protected Object convertConstant(org.apache.iceberg.types.Type type, Object value) {
     return GenericDataUtil.internalToGeneric(type, value);
   }
 
-  private static class RecordReader extends StructReader<Record, Record> {
-    private final GenericRecord template;
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
 
-    RecordReader(List<Type> types, List<ParquetValueReader<?>> readers, StructType struct) {
-      super(types, readers);
-      this.template = struct != null ? GenericRecord.create(struct) : null;
+  protected static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {
+    protected DateReader(ColumnDescriptor desc) {
+      super(desc);
     }
 
     @Override
-    protected Record newStructData(Record reuse) {
-      if (reuse != null) {
-        return reuse;
-      } else {
-        // GenericRecord.copy() is more performant then GenericRecord.create(StructType) since
-        // NAME_MAP_CACHE access
-        // is eliminated. Using copy here to gain performance.
-        return template.copy();
-      }
+    public LocalDate read(LocalDate reuse) {
+      return EPOCH_DAY.plusDays(column.nextInteger());
+    }
+  }
+
+  protected static class TimestampReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    protected TimestampReader(ColumnDescriptor desc) {
+      super(desc);
     }
 
     @Override
-    protected Object getField(Record intermediate, int pos) {
-      return intermediate.get(pos);
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  protected static class TimestampMillisReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    protected TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
     }
 
     @Override
-    protected Record buildStruct(Record struct) {
-      return struct;
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  protected static class TimestampInt96Reader

Review Comment:
   I think this and the other classes should be package-private rather than protected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925922076


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -192,13 +222,17 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      return Optional.ofNullable(dateWriter(desc));
     }
 
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeType) {
-      return Optional.of(new TimeWriter(desc));
+      Preconditions.checkArgument(
+          LogicalTypeAnnotation.TimeUnit.MICROS.equals(timeType.getUnit()),
+          "Cannot write time in %s, only MICROS is supported",
+          timeType.getUnit());
+      return Optional.ofNullable(timeWriter(desc));

Review Comment:
   I don't think there is a need for a pass-through. That behavior isn't very easy to follow if you're trying to understand what this is doing. Instead, it's better to always implement `timeWriter` and others.



##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -190,33 +210,49 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     }
 
     @Override
-    public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
+    public Optional<ParquetValueWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      ParquetValueWriter<?> dateWriter = dateWriter(desc);
+      if (dateWriter != null) {
+        return Optional.of(dateWriter(desc));
+      }
+
+      return LogicalTypeAnnotation.LogicalTypeAnnotationVisitor.super.visit(dateType);

Review Comment:
   Is your intent to not support dates, times, or timestamps for the internal model? If so, why?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925927584


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetWriter.java:
##########
@@ -35,17 +46,75 @@ public static ParquetValueWriter<Record> buildWriter(MessageType type) {
 
   @Override
   protected StructWriter<Record> createStructWriter(List<ParquetValueWriter<?>> writers) {
-    return new RecordWriter(writers);
+    return ParquetValueWriters.recordWriter(writers);
   }
 
-  private static class RecordWriter extends StructWriter<Record> {
-    private RecordWriter(List<ParquetValueWriter<?>> writers) {
-      super(writers);
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  protected static class DateWriter extends ParquetValueWriters.PrimitiveWriter<LocalDate> {

Review Comment:
   Similar, these should be package-private.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1926276720


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new DateWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new TimeWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new TimestamptzWriter(desc);
+    } else {
+      return new TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer; no special writer needed.

Review Comment:
   hmm,
   Only Arrow has the testcase for UUID, other usage of generic writers doesn't cover UUID data type!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1926270438


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -850,4 +919,42 @@ private TripleIterator<?> firstNonNullColumn(List<TripleIterator<?>> columns) {
       return NullReader.NULL_COLUMN;
     }
   }
+
+  private static class RecordReader<T extends StructLike> extends StructReader<T, T> {

Review Comment:
   Internal readers expects `ParquetValueReader<StructLike>` and generic reader expects `ParquetValueReader<Record>`, even though the `Record` extends `StructLike`, because of java generics, we need to return the specific type here. We cannot cast `ParquetValueReader<Record>` to `ParquetValueReader<StructLike>`. 
   
   Hence, this was refactored. Similar to the writer refactoring.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906134768


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {
+    private final GenericRecord template;
+
+    ParquetStructReader(List<Type> types, List<ParquetValueReader<?>> readers, StructType struct) {
+      super(types, readers);
+      this.template = struct != null ? GenericRecord.create(struct) : null;
+    }
+
+    @Override
+    protected StructLike newStructData(StructLike reuse) {
+      if (reuse != null) {
+        return reuse;
+      } else {
+        return template.copy();
+      }
+    }
+
+    @Override
+    protected Object getField(StructLike intermediate, int pos) {
+      return intermediate.get(pos, Object.class);
+    }
+
+    @Override
+    protected StructLike buildStruct(StructLike struct) {
+      return struct;
+    }
+
+    @Override
+    protected void set(StructLike struct, int pos, Object value) {
+      struct.set(pos, value);
+    }
+  }
+
+  private static class LogicalTypeAnnotationParquetValueReaderVisitor
+      implements LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>> {
+
+    private final ColumnDescriptor desc;
+    private final org.apache.iceberg.types.Type.PrimitiveType expected;
+    private final PrimitiveType primitive;
+
+    LogicalTypeAnnotationParquetValueReaderVisitor(
+        ColumnDescriptor desc,
+        org.apache.iceberg.types.Type.PrimitiveType expected,
+        PrimitiveType primitive) {
+      this.desc = desc;
+      this.expected = expected;
+      this.primitive = primitive;
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.StringLogicalTypeAnnotation stringLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.EnumLogicalTypeAnnotation enumLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.UUIDLogicalTypeAnnotation uuidLogicalType) {
+      return Optional.of(new ParquetValueReaders.UUIDReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalLogicalType) {
+      switch (primitive.getPrimitiveTypeName()) {
+        case BINARY:
+        case FIXED_LEN_BYTE_ARRAY:
+          return Optional.of(
+              new ParquetValueReaders.BinaryAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT64:
+          return Optional.of(
+              new ParquetValueReaders.LongAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT32:
+          return Optional.of(
+              new ParquetValueReaders.IntegerAsDecimalReader(desc, decimalLogicalType.getScale()));
+        default:
+          throw new UnsupportedOperationException(
+              "Unsupported base type for decimal: " + primitive.getPrimitiveTypeName());
+      }
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DateLogicalTypeAnnotation dateLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestampLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));

Review Comment:
   This isn't correct. The unit of the incoming timestamp value still needs to be handled, even if the in-memory representation of the value is the same (a `long`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906132346


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +64,16 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected abstract LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>

Review Comment:
   I don't think it makes sense to have the subclasses provide this visitor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1911784070


##########
.palantir/revapi.yml:
##########
@@ -1171,6 +1171,28 @@ acceptedBreaks:
         \ java.util.function.Function<T, java.lang.Long>, org.apache.iceberg.io.CloseableIterable<java.lang.Long>,\
         \ java.util.function.Consumer<T>)"
       justification: "Removing deprecated code"
+    org.apache.iceberg:iceberg-parquet:
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueReaders.PrimitiveReader<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::fixedReader(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueReaders.PrimitiveReader<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::int96Reader(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueWriters.PrimitiveWriter<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetWriter<T>::fixedWriter(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.parquet.schema.LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<org.apache.iceberg.parquet.ParquetValueReader<?>>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::logicalTypeReaderVisitor(org.apache.parquet.column.ColumnDescriptor,\
+        \ org.apache.iceberg.types.Type.PrimitiveType, org.apache.parquet.schema.PrimitiveType)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"

Review Comment:
   Oh, I think what you mean is don't add it as abstract. Add it as methods with default implementation. I got it. I will update it today. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917496847


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Optional;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.PrimitiveWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> buildWriter(MessageType type) {

Review Comment:
   I think this would be better as `create` as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917277358


##########
data/src/test/java/org/apache/iceberg/data/parquet/TestInternalData.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.InternalTestHelpers;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RandomInternalData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.DataTest;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+public class TestInternalData extends DataTest {

Review Comment:
   Why not use `AvroDataTest` that is provided by the core tests module and is used for `TestInternalAvro`?
   
   That works for me locally.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917263968


##########
api/src/test/java/org/apache/iceberg/util/RandomUtil.java:
##########
@@ -237,7 +237,7 @@ private static BigInteger randomUnscaled(int precision, Random random) {
   }
 
   public static List<Object> generateList(
-      Random random, Types.ListType list, Supplier<Object> elementResult) {

Review Comment:
   How about `elementSupplier` instead? I think that's more accurate than `elements` for two reasons. First, the argument is not an element, it is a function that produces an element. Second, there is only one supplier so the name shouldn't be plural.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917278339


##########
data/src/test/java/org/apache/iceberg/data/parquet/TestInternalData.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.InternalTestHelpers;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RandomInternalData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.DataTest;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+public class TestInternalData extends DataTest {
+  @Override
+  protected void writeAndValidate(Schema schema) throws IOException {
+    writeAndValidate(schema, schema);
+  }
+
+  @Override
+  protected void writeAndValidate(Schema writeSchema, Schema expectedSchema) throws IOException {
+    List<StructLike> expected = RandomInternalData.generate(writeSchema, 100, 42L);
+
+    File testFile = File.createTempFile("junit", null, temp.toFile());
+    assertThat(testFile.delete()).isTrue();
+
+    OutputFile outputFile = Files.localOutput(testFile);

Review Comment:
   As with the Avro equivalent, I think this should use `new InMemoryOutputFile()` to avoid using a temp directory and make the tests faster.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917299961


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timeReader(ColumnDescriptor desc, TimeUnit unit) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, TimeUnit unit, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {

Review Comment:
   Why is there a factory method for UUID? Both uses of this class have the same representation so a new method is not required.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925697904


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -192,13 +222,17 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      return Optional.ofNullable(dateWriter(desc));
     }
 
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeType) {
-      return Optional.of(new TimeWriter(desc));
+      Preconditions.checkArgument(
+          LogicalTypeAnnotation.TimeUnit.MICROS.equals(timeType.getUnit()),
+          "Cannot write time in %s, only MICROS is supported",
+          timeType.getUnit());
+      return Optional.ofNullable(timeWriter(desc));

Review Comment:
   Because the `InternalWriter` returns null for `timewriter()` meaning use the primitive writer (no need of special handling). Existing `Optional.of()` will give NPE. To avoid it, I used  `Optional.ofNullable()` so it returns `Optional.empty()` when it is null. 
   
   It was also the reason why I was returning optional from the default implementation signatures in https://github.com/apache/iceberg/pull/11904#discussion_r1917303018. 
   
   I will do the null check manually and return super method and revert back to `Optional.of()`, if it looks odd. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925912073


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> create(MessageType type) {
+    return INSTANCE.createWriter(type);
+  }
+
+  @Override
+  protected StructWriter<StructLike> createStructWriter(List<ParquetValueWriter<?>> writers) {
+    return ParquetValueWriters.recordWriter(writers);
+  }
+
+  @Override
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return ParquetValueWriters.fixedBuffers(desc);
+  }
+
+  @Override
+  protected ParquetValueWriter<?> dateWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer; no special writer needed.
+    return null;

Review Comment:
   All of these methods should return the correct writer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924389663


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -92,4 +151,124 @@ protected void set(Record struct, int pos, Object value) {
       struct.set(pos, value);
     }
   }
+
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  private static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {
+    private DateReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDate read(LocalDate reuse) {
+      return EPOCH_DAY.plusDays(column.nextInteger());
+    }
+  }
+
+  private static class TimestampReader extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampMillisReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampInt96Reader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private static final long UNIX_EPOCH_JULIAN = 2_440_588L;
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      final long timeOfDayNanos = byteBuffer.getLong();
+      final int julianDay = byteBuffer.getInt();
+
+      return Instant.ofEpochMilli(TimeUnit.DAYS.toMillis(julianDay - UNIX_EPOCH_JULIAN))
+          .plusNanos(timeOfDayNanos)
+          .atOffset(ZoneOffset.UTC);
+    }
+  }
+
+  private static class TimestamptzReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimestamptzMillisReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimeMillisReader extends ParquetValueReaders.PrimitiveReader<LocalTime> {
+    private TimeMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalTime read(LocalTime reuse) {
+      return LocalTime.ofNanoOfDay(column.nextLong() * 1000000L);

Review Comment:
   Yes, I meant `nextInteger`. Thanks for fixing this.
   
   No need to modify the tests in this PR. It would be nice to have a test for millis, but these were written based on the spec originally and we don't need to block to add a test for existing code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Internal writer and reader [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat closed pull request #11904: Parquet: Internal writer and reader
URL: https://github.com/apache/iceberg/pull/11904


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Internal writer and reader [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1901970283


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {
+    private final GenericRecord template;

Review Comment:
   Should I introduce a new class like `GenericStruct` that implements `StructLike`, I thought of using `StructProjection` but it is read only. Hence, need one simple class to create a `StructLike` objects. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925113427


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {

Review Comment:
   Existing class `LogicalTypeWriterVisitor` have declared return values as `Optional<ParquetValueWriters.PrimitiveWriter<?>>`. So, I need to modify the base class also to return `ParquetValueWriter<?>`. 
   
   I thought I will get a comment that why we modified base class and avoid unnecessary refactoring!
   Hence, I didn't do. 
   
   Since, you also want to handle this. I will update it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927713133


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {

Review Comment:
   I opened a PR with these changes to make it easier.



##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -412,6 +446,41 @@ public byte[] read(byte[] ignored) {
     }
   }
 
+  private static class TimestampInt96Reader extends UnboxedReader<Long> {
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      return ParquetUtil.extractTimestampInt96(byteBuffer);
+    }
+  }
+
+  private static class TimestampMillisReader extends UnboxedReader<Long> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      return 1000L * column.nextInteger();

Review Comment:
   Fixed in https://github.com/ajantha-bhat/iceberg/pull/74



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927628058


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<Record> create(Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<Record> create(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<Record> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return ParquetValueReaders.recordReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected ParquetValueReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReader<?> dateReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  @Override
+  protected ParquetValueReader<?> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return ParquetValueReaders.millisAsTimestamps(desc);

Review Comment:
   This converts micros to millis, but the problem is that it first reads using `nextLong` when we expect an int. Looks like there was some confusion over this because the `timestamp` reader (see my comment below) now calls `nextInteger` when it should continue to use `nextLong`).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Internal writer and reader [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1901971027


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestInternalWriter.java:
##########
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.apache.iceberg.parquet.ParquetWritingTestUtils.createTempFile;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.UUID;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.data.parquet.InternalReader;
+import org.apache.iceberg.data.parquet.InternalWriter;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructProjection;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestInternalWriter {
+  @TempDir private Path temp;
+
+  @Test
+  public void testDataWriter() throws IOException {
+    Schema schema =
+        new Schema(
+            required(100, "b", Types.BooleanType.get()),
+            optional(101, "i", Types.IntegerType.get()),
+            required(102, "l", Types.LongType.get()),
+            optional(103, "f", Types.FloatType.get()),
+            required(104, "d", Types.DoubleType.get()),
+            optional(105, "date", Types.DateType.get()),
+            required(106, "time", Types.TimeType.get()),
+            required(107, "ts", Types.TimestampType.withoutZone()),
+            required(108, "ts_tz", Types.TimestampType.withZone()),
+            required(109, "s", Types.StringType.get()),
+            required(110, "uuid", Types.UUIDType.get()),
+            required(111, "fixed", Types.FixedType.ofLength(7)),
+            optional(112, "bytes", Types.BinaryType.get()),
+            required(113, "dec_38_10", Types.DecimalType.of(38, 10)));
+
+    // Consuming the data as per Type.java
+    GenericRecord record = GenericRecord.create(schema);

Review Comment:
   Should I introduce a new class like `GenericStruct` that implements `StructLike`, I thought of using `StructProjection` but it is read only. Hence, need one simple class to create a `StructLike` objects. 
   
   Line 95, wrapping this with `StructProjection` just to get pure `StructLike`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917518515


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   I didn't do it because I can't move the date/time classes to `GenericReader` in that case. It has to be in this class. But yeah, I got your point about behavior change.  I will revert that date/time classes movement. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1926371943


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +46,31 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.FixedWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.DateWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.TimeWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timestampWriter(ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new GenericParquetWriter.TimestamptzWriter(desc);
+    } else {
+      return new GenericParquetWriter.TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer (as FIXED_LEN_BYTE_ARRAY); no special writer needed.
+    return null;

Review Comment:
   Sorry, the existing testcases was the reason for confusion as I mentioned in https://github.com/apache/iceberg/pull/11904#discussion_r1926274339
   
   I will update the existing testcases of Arrow too in this PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927618131


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new DateWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new TimeWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new TimestamptzWriter(desc);
+    } else {
+      return new TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer; no special writer needed.

Review Comment:
   Okay, I did some digging and it looks like this behavior was accidentally introduced when moving to the type annotation visitor (https://github.com/apache/iceberg/commit/1d13a10d4c8a1174083eb81373310b8499f84f44). Before that point, UUID was not supported but after that it would fall through to the primitive type and return a writer for bytes.
   
   The problem is that this conflicts with the representation used by the Avro generic object model: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/avro/ValueReaders.java#L108-L110
   
   I think this change is the right choice because the object models should match and using UUID for this is well-established in the Avro implementation and the generic random data generator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925905904


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +46,31 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.FixedWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.DateWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.TimeWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timestampWriter(ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new GenericParquetWriter.TimestamptzWriter(desc);
+    } else {
+      return new GenericParquetWriter.TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer (as FIXED_LEN_BYTE_ARRAY); no special writer needed.
+    return null;

Review Comment:
   I think I commented on this in the last round of reviews. This isn't correct. Incoming values are of type `UUID` so this needs a writer that can convert UUID into a byte array. This should return `ParquetValueWriter.uuids(desc)`.
   
   There's also no need to add a method for this because it is the same between the generic and internal object models.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1926274339


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new DateWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new TimeWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new TimestamptzWriter(desc);
+    } else {
+      return new TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer; no special writer needed.

Review Comment:
   All the generic reader writer test case uses UUID as byte[], hence it is working. 
   
   You mean existing testcases are wrong?
   https://github.com/apache/iceberg/blob/be6e9daf8901cdee63197e77fcf95624bb694f39/arrow/src/test/java/org/apache/iceberg/arrow/vectorized/ArrowReaderTest.java#L861-L862



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927628058


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<Record> create(Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<Record> create(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<Record> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return ParquetValueReaders.recordReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected ParquetValueReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReader<?> dateReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  @Override
+  protected ParquetValueReader<?> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return ParquetValueReaders.millisAsTimestamps(desc);

Review Comment:
   This isn't correct, but it does match the current behavior so it isn't a blocker. This converts micros to millis, but the problem is that it first reads using `getLong` when we expect an int. Since the other time millis reader was also broken, I think we can fix this in a follow up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924346731


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -63,6 +69,18 @@ public static ParquetValueReader<Long> position() {
     return new PositionReader();
   }
 
+  public static ParquetValueReader<UUID> uuids(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UUIDReader(desc);
+  }
+
+  public static ParquetValueReader<Long> timestampInt96Reader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampInt96Reader(desc);
+  }
+
+  public static ParquetValueReader<Long> timestampMillisReader(ColumnDescriptor desc) {

Review Comment:
   I'd probably rename this as well, maybe `milliTimestamps` or `millisecondTimestamps`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917279635


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/data/SparkParquetReaders.java:
##########
@@ -373,41 +371,6 @@ public Decimal read(Decimal ignored) {
     }
   }
 
-  private static class TimestampMillisReader extends UnboxedReader<Long> {

Review Comment:
   I think this is fine. No need to change older Spark versions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917303932


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {

Review Comment:
   These should all return `ParquetValueReader<?>` rather than constrain the implementation to a subclass of `PrimitiveReader`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917325058


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueWriters.java:
##########
@@ -330,6 +382,17 @@ public void write(int repetitionLevel, CharSequence value) {
     }
   }
 
+  private static class UUIDWriter extends PrimitiveWriter<UUID> {
+    private UUIDWriter(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public void write(int repetitionLevel, UUID value) {
+      column.writeBinary(repetitionLevel, Binary.fromReusedByteArray(UUIDUtil.convert(value)));

Review Comment:
   The byte array passed here is allocated, not reused. It probably should be reused, like the one in Spark's writers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906132097


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -114,113 +112,6 @@ public ParquetValueReader<?> struct(
     }
   }
 
-  private class LogicalTypeAnnotationParquetValueReaderVisitor
-      implements LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>> {
-
-    private final ColumnDescriptor desc;
-    private final org.apache.iceberg.types.Type.PrimitiveType expected;
-    private final PrimitiveType primitive;
-
-    LogicalTypeAnnotationParquetValueReaderVisitor(
-        ColumnDescriptor desc,
-        org.apache.iceberg.types.Type.PrimitiveType expected,
-        PrimitiveType primitive) {
-      this.desc = desc;
-      this.expected = expected;
-      this.primitive = primitive;
-    }
-
-    @Override
-    public Optional<ParquetValueReader<?>> visit(
-        LogicalTypeAnnotation.StringLogicalTypeAnnotation stringLogicalType) {
-      return Optional.of(new ParquetValueReaders.StringReader(desc));
-    }
-
-    @Override
-    public Optional<ParquetValueReader<?>> visit(
-        LogicalTypeAnnotation.EnumLogicalTypeAnnotation enumLogicalType) {
-      return Optional.of(new ParquetValueReaders.StringReader(desc));
-    }
-
-    @Override
-    public Optional<ParquetValueReader<?>> visit(DecimalLogicalTypeAnnotation decimalLogicalType) {
-      switch (primitive.getPrimitiveTypeName()) {
-        case BINARY:
-        case FIXED_LEN_BYTE_ARRAY:
-          return Optional.of(
-              new ParquetValueReaders.BinaryAsDecimalReader(desc, decimalLogicalType.getScale()));
-        case INT64:
-          return Optional.of(
-              new ParquetValueReaders.LongAsDecimalReader(desc, decimalLogicalType.getScale()));
-        case INT32:
-          return Optional.of(
-              new ParquetValueReaders.IntegerAsDecimalReader(desc, decimalLogicalType.getScale()));
-        default:
-          throw new UnsupportedOperationException(
-              "Unsupported base type for decimal: " + primitive.getPrimitiveTypeName());
-      }
-    }
-
-    @Override
-    public Optional<ParquetValueReader<?>> visit(
-        LogicalTypeAnnotation.DateLogicalTypeAnnotation dateLogicalType) {
-      return Optional.of(new DateReader(desc));

Review Comment:
   This and the following 2 methods are the only changes between the implementations of this class, so a lot of code is duplicated. In addition, this already introduces abstract factory methods for some readers -- including timestamps. I think it would be much cleaner to reuse this and call factory methods instead:
   
   ```java
     protected abstract PrimitiveReader<?> dateReader(ColumnDescriptor desc);
     protected abstract PrimitiveReader<?> timeReader(ColumnDescriptor desc, ChronoUnit unit);
     protected abstract PrimitiveReader<?> timestampReader(ColumnDescriptor desc, ChronoUnit unit, boolean isAdjustedToUTC);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906067488


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Optional;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> buildWriter(MessageType type) {
+    return INSTANCE.createWriter(type);
+  }
+
+  @Override
+  protected StructWriter<StructLike> createStructWriter(List<ParquetValueWriter<?>> writers) {
+    return new ParquetStructWriter(writers);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<
+          ParquetValueWriters.PrimitiveWriter<?>>
+      logicalTypeWriterVisitor(ColumnDescriptor desc) {
+    return new LogicalTypeWriterVisitor(desc);
+  }
+
+  @Override
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    // accepts ByteBuffer and internally writes as binary.
+    return ParquetValueWriters.byteBuffers(desc);
+  }
+
+  private static class ParquetStructWriter extends StructWriter<StructLike> {

Review Comment:
   I think this should be `StructLikeWriter`. This is in the `parquet` package and there isn't much value to adding it to this name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906143562


##########
.palantir/revapi.yml:
##########
@@ -1171,6 +1171,28 @@ acceptedBreaks:
         \ java.util.function.Function<T, java.lang.Long>, org.apache.iceberg.io.CloseableIterable<java.lang.Long>,\
         \ java.util.function.Consumer<T>)"
       justification: "Removing deprecated code"
+    org.apache.iceberg:iceberg-parquet:
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueReaders.PrimitiveReader<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::fixedReader(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueReaders.PrimitiveReader<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::int96Reader(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueWriters.PrimitiveWriter<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetWriter<T>::fixedWriter(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.parquet.schema.LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<org.apache.iceberg.parquet.ParquetValueReader<?>>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::logicalTypeReaderVisitor(org.apache.parquet.column.ColumnDescriptor,\
+        \ org.apache.iceberg.types.Type.PrimitiveType, org.apache.parquet.schema.PrimitiveType)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"

Review Comment:
   This PR should not introduce revapi failures. Instead, the new methods should have default implementations that match the previous behavior (returning the generic representations).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Internal writer and reader [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1905999633


##########
.palantir/revapi.yml:
##########
@@ -1171,6 +1171,28 @@ acceptedBreaks:
         \ java.util.function.Function<T, java.lang.Long>, org.apache.iceberg.io.CloseableIterable<java.lang.Long>,\
         \ java.util.function.Consumer<T>)"
       justification: "Removing deprecated code"
+    org.apache.iceberg:iceberg-parquet:
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueReaders.PrimitiveReader<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::fixedReader(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"

Review Comment:
   Why are there curly braces in the justification text?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906016284


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -401,6 +403,17 @@ public ByteBuffer read(ByteBuffer reuse) {
     }
   }
 
+  public static class UUIDReader extends PrimitiveReader<UUID> {
+    public UUIDReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public UUID read(UUID reuse) {
+      return UUIDUtil.convert(column.nextBinary().toByteBuffer());

Review Comment:
   This looks fine to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906141975


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestInternalWriter.java:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.parquet;
+
+import static org.apache.iceberg.parquet.ParquetWritingTestUtils.createTempFile;
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.Comparator;
+import java.util.List;
+import java.util.UUID;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileContent;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.parquet.InternalReader;
+import org.apache.iceberg.data.parquet.InternalWriter;
+import org.apache.iceberg.expressions.Literal;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Comparators;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructProjection;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TestInternalWriter {

Review Comment:
   As with the Avro tests, I think this should extend `DataTest`. It is probably easier to do the Avro work first and then reuse it here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906125908


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -359,10 +250,10 @@ public ParquetValueReader<?> primitive(
 
       ColumnDescriptor desc = type.getColumnDescription(currentPath());
 
-      if (primitive.getOriginalType() != null) {
+      if (primitive.getLogicalTypeAnnotation() != null) {

Review Comment:
   I agree with this change, but please point these kinds of changes out for reviewers.
   
   The old version worked because all of the supported logical type annotations had an equivalent `ConvertedType` (which is what `OriginalType` is called in Parquet format and the [logical type docs](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#numeric-types)).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918344913


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -92,4 +151,124 @@ protected void set(Record struct, int pos, Object value) {
       struct.set(pos, value);
     }
   }
+
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  private static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {
+    private DateReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDate read(LocalDate reuse) {
+      return EPOCH_DAY.plusDays(column.nextInteger());
+    }
+  }
+
+  private static class TimestampReader extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampMillisReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampInt96Reader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private static final long UNIX_EPOCH_JULIAN = 2_440_588L;
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      final long timeOfDayNanos = byteBuffer.getLong();
+      final int julianDay = byteBuffer.getInt();
+
+      return Instant.ofEpochMilli(TimeUnit.DAYS.toMillis(julianDay - UNIX_EPOCH_JULIAN))
+          .plusNanos(timeOfDayNanos)
+          .atOffset(ZoneOffset.UTC);
+    }
+  }
+
+  private static class TimestamptzReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimestamptzMillisReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimeMillisReader extends ParquetValueReaders.PrimitiveReader<LocalTime> {
+    private TimeMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalTime read(LocalTime reuse) {
+      return LocalTime.ofNanoOfDay(column.nextLong() * 1000000L);

Review Comment:
   > so I think this should call column.nextLong instead.
   You mean `column.nextInt` right?
   
   So the existing code is wrong? I didn't added this code newly.
   how do I test these? `DataTest` doesn't handle time units and writers write in micro always? 
   



##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -92,4 +151,124 @@ protected void set(Record struct, int pos, Object value) {
       struct.set(pos, value);
     }
   }
+
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  private static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {
+    private DateReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDate read(LocalDate reuse) {
+      return EPOCH_DAY.plusDays(column.nextInteger());
+    }
+  }
+
+  private static class TimestampReader extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampMillisReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampInt96Reader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private static final long UNIX_EPOCH_JULIAN = 2_440_588L;
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      final long timeOfDayNanos = byteBuffer.getLong();
+      final int julianDay = byteBuffer.getInt();
+
+      return Instant.ofEpochMilli(TimeUnit.DAYS.toMillis(julianDay - UNIX_EPOCH_JULIAN))
+          .plusNanos(timeOfDayNanos)
+          .atOffset(ZoneOffset.UTC);
+    }
+  }
+
+  private static class TimestamptzReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimestamptzMillisReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimeMillisReader extends ParquetValueReaders.PrimitiveReader<LocalTime> {
+    private TimeMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalTime read(LocalTime reuse) {
+      return LocalTime.ofNanoOfDay(column.nextLong() * 1000000L);

Review Comment:
   > so I think this should call column.nextLong instead.
   
   You mean `column.nextInt` right?
   
   So the existing code is wrong? I didn't added this code newly.
   how do I test these? `DataTest` doesn't handle time units and writers write in micro always? 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917320632


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -401,6 +404,17 @@ public ByteBuffer read(ByteBuffer reuse) {
     }
   }
 
+  public static class UUIDReader extends PrimitiveReader<UUID> {

Review Comment:
   Rather than making the classes here public, I would prefer to introduce factory methods like we have for the Avro readers. I think that gives us more flexibility to change implementations and move classes around without worrying because none of them need to be public.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917518515


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   I didn't do it because, I can't move the date/time classes to `GenericReader`. It has to be in this class. But yeah, I got your point about behavior change. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918350294


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new StructLikeReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.TimestampInt96Reader(desc);
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return Optional.of(new ParquetValueReaders.TimestampMillisReader(desc));
+    }
+
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit, boolean isAdjustedToUTC) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return Optional.of(new ParquetValueReaders.TimestampMillisReader(desc));
+    }
+
+    return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+  }
+
+  @Override
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {
+    return Optional.of(new ParquetValueReaders.UUIDReader(desc));
+  }
+
+  private static class StructLikeReader extends StructReader<StructLike, StructLike> {
+    private final GenericRecord template;
+
+    StructLikeReader(List<Type> types, List<ParquetValueReader<?>> readers, StructType struct) {
+      super(types, readers);

Review Comment:
   It touches Flink and other classes. I will handle in separate PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1908250543


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {

Review Comment:
   Cannot reuse the class from `GenericParquetReaders` as  it is based on `Record` interface, we need a class based on `StructLike` interface. 
   
   I have renamed to `StructLikeReader`, just like the `StructLikeWriter` from `InternalWriter` class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924830895


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -239,6 +269,12 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.BsonLogicalTypeAnnotation bsonType) {
       return Optional.of(ParquetValueWriters.byteBuffers(desc));
     }
+
+    @Override
+    public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
+        LogicalTypeAnnotation.UUIDLogicalTypeAnnotation uuidLogicalType) {
+      return Optional.ofNullable(uuidWriter(desc));

Review Comment:
   Generic writers handle UUID logical type by retuning default empty(), so that it will be handled as primitive writer of fixed length byte[] here 
   https://github.com/apache/iceberg/blob/5b13760c01bdbb2ab15be072c582adb2a8792f23/parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java#L131C1-L132C1
   
   But internal writers needs new UUID based writer. Hence, I added an override to return empty writer for generic types (default) and UUID based writer for internal type. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925912623


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -190,33 +210,49 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     }
 
     @Override
-    public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
+    public Optional<ParquetValueWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      ParquetValueWriter<?> dateWriter = dateWriter(desc);
+      if (dateWriter != null) {
+        return Optional.of(dateWriter(desc));
+      }
+
+      return LogicalTypeAnnotation.LogicalTypeAnnotationVisitor.super.visit(dateType);

Review Comment:
   Is your intent to not support dates for the internal model? If so, why?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue merged PR #11904:
URL: https://github.com/apache/iceberg/pull/11904


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925106066


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new DateWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new TimeWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new TimestamptzWriter(desc);
+    } else {
+      return new TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer; no special writer needed.

Review Comment:
   It uses the fixed length primitive writer with byte[] as input. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924348066


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueWriters.java:
##########
@@ -106,6 +114,10 @@ public static PrimitiveWriter<ByteBuffer> byteBuffers(ColumnDescriptor desc) {
     return new BytesWriter(desc);
   }
 
+  public static PrimitiveWriter<ByteBuffer> fixedBuffer(ColumnDescriptor desc) {

Review Comment:
   Most other methods are plural. How about `fixedBuffers`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924384924


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -397,7 +404,7 @@ public ParquetValueReader<?> primitive(
         case INT96:
           // Impala & Spark used to write timestamps as INT96 without a logical type. For backwards
           // compatibility we try to read INT96 as timestamps.
-          return new TimestampInt96Reader(desc);
+          return int96Reader(desc);

Review Comment:
   Yes. Because the reader produces and `OffsetDateTime`, it should only be used when the expected type is compatible.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925726162


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   I see. I moved them back now. I didn't create extra factory methods for these existing classes after the movement as I felt it is an overkill and behavior shouldn't change for these generic readers and writers.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925907570


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -190,33 +210,49 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     }
 
     @Override
-    public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
+    public Optional<ParquetValueWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      ParquetValueWriter<?> dateWriter = dateWriter(desc);
+      if (dateWriter != null) {

Review Comment:
   There is no need to handle nulls here. A writer should always be returned by the factory methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1908952177


##########
.palantir/revapi.yml:
##########
@@ -1171,6 +1171,28 @@ acceptedBreaks:
         \ java.util.function.Function<T, java.lang.Long>, org.apache.iceberg.io.CloseableIterable<java.lang.Long>,\
         \ java.util.function.Consumer<T>)"
       justification: "Removing deprecated code"
+    org.apache.iceberg:iceberg-parquet:
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueReaders.PrimitiveReader<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::fixedReader(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueReaders.PrimitiveReader<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::int96Reader(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.iceberg.parquet.ParquetValueWriters.PrimitiveWriter<?>\
+        \ org.apache.iceberg.data.parquet.BaseParquetWriter<T>::fixedWriter(org.apache.parquet.column.ColumnDescriptor)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"
+      new: "method org.apache.parquet.schema.LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<org.apache.iceberg.parquet.ParquetValueReader<?>>\
+        \ org.apache.iceberg.data.parquet.BaseParquetReaders<T>::logicalTypeReaderVisitor(org.apache.parquet.column.ColumnDescriptor,\
+        \ org.apache.iceberg.types.Type.PrimitiveType, org.apache.parquet.schema.PrimitiveType)"
+      justification: "{Refactor Parquet reader and writer}"
+    - code: "java.method.abstractMethodAdded"

Review Comment:
   New methods are abstract and abstract method cannot have default implementation. So, I think we have to handle revapi failures. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906139217


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {
+    private final GenericRecord template;
+
+    ParquetStructReader(List<Type> types, List<ParquetValueReader<?>> readers, StructType struct) {
+      super(types, readers);
+      this.template = struct != null ? GenericRecord.create(struct) : null;
+    }
+
+    @Override
+    protected StructLike newStructData(StructLike reuse) {
+      if (reuse != null) {
+        return reuse;
+      } else {
+        return template.copy();
+      }
+    }
+
+    @Override
+    protected Object getField(StructLike intermediate, int pos) {
+      return intermediate.get(pos, Object.class);
+    }
+
+    @Override
+    protected StructLike buildStruct(StructLike struct) {
+      return struct;
+    }
+
+    @Override
+    protected void set(StructLike struct, int pos, Object value) {
+      struct.set(pos, value);
+    }
+  }
+
+  private static class LogicalTypeAnnotationParquetValueReaderVisitor
+      implements LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>> {
+
+    private final ColumnDescriptor desc;
+    private final org.apache.iceberg.types.Type.PrimitiveType expected;
+    private final PrimitiveType primitive;
+
+    LogicalTypeAnnotationParquetValueReaderVisitor(
+        ColumnDescriptor desc,
+        org.apache.iceberg.types.Type.PrimitiveType expected,
+        PrimitiveType primitive) {
+      this.desc = desc;
+      this.expected = expected;
+      this.primitive = primitive;
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.StringLogicalTypeAnnotation stringLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.EnumLogicalTypeAnnotation enumLogicalType) {
+      return Optional.of(new ParquetValueReaders.StringReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.UUIDLogicalTypeAnnotation uuidLogicalType) {
+      return Optional.of(new ParquetValueReaders.UUIDReader(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DecimalLogicalTypeAnnotation decimalLogicalType) {
+      switch (primitive.getPrimitiveTypeName()) {
+        case BINARY:
+        case FIXED_LEN_BYTE_ARRAY:
+          return Optional.of(
+              new ParquetValueReaders.BinaryAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT64:
+          return Optional.of(
+              new ParquetValueReaders.LongAsDecimalReader(desc, decimalLogicalType.getScale()));
+        case INT32:
+          return Optional.of(
+              new ParquetValueReaders.IntegerAsDecimalReader(desc, decimalLogicalType.getScale()));
+        default:
+          throw new UnsupportedOperationException(
+              "Unsupported base type for decimal: " + primitive.getPrimitiveTypeName());
+      }
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.DateLogicalTypeAnnotation dateLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));
+    }
+
+    @Override
+    public Optional<ParquetValueReader<?>> visit(
+        LogicalTypeAnnotation.TimestampLogicalTypeAnnotation timestampLogicalType) {
+      return Optional.of(new ParquetValueReaders.UnboxedReader<>(desc));

Review Comment:
   Looks like the Spark implementations for this should work well, just like the int96 cases.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#issuecomment-2613443297

   Thanks, @ajantha-bhat! Good to get this in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924348066


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueWriters.java:
##########
@@ -106,6 +114,10 @@ public static PrimitiveWriter<ByteBuffer> byteBuffers(ColumnDescriptor desc) {
     return new BytesWriter(desc);
   }
 
+  public static PrimitiveWriter<ByteBuffer> fixedBuffer(ColumnDescriptor desc) {

Review Comment:
   Minor: Most other methods are plural. How about `fixedBuffers`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924342038


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -239,6 +269,12 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.BsonLogicalTypeAnnotation bsonType) {
       return Optional.of(ParquetValueWriters.byteBuffers(desc));
     }
+
+    @Override
+    public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
+        LogicalTypeAnnotation.UUIDLogicalTypeAnnotation uuidLogicalType) {
+      return Optional.ofNullable(uuidWriter(desc));

Review Comment:
   I don't think there's any need to make this more flexible. It makes sense to support the new `UUIDWriter` to ensure the tests pass with all primitive types, but this doesn't need to introduce a way to override the writer and representation (same for the read side).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925925464


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {

Review Comment:
   Returning the right type (in this case, the interface) is a good thing. It's worth refactoring the private visitor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927638068


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -412,6 +446,41 @@ public byte[] read(byte[] ignored) {
     }
   }
 
+  private static class TimestampInt96Reader extends UnboxedReader<Long> {
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      return ParquetUtil.extractTimestampInt96(byteBuffer);
+    }
+  }
+
+  private static class TimestampMillisReader extends UnboxedReader<Long> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public Long read(Long ignored) {
+      return readLong();
+    }
+
+    @Override
+    public long readLong() {
+      return 1000L * column.nextInteger();

Review Comment:
   I think the confusion was from this comment: https://github.com/apache/iceberg/pull/11904#discussion_r1906138972
   
   I was talking about the time type, but the code I pasted had the wrong class name, `TimestampMillisReader` should have been `TimeMillisReader`. Timestamps (millis) should use `nextLong` and time (millis) should use `nextInteger`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1927639914


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.data.Record;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<Record> {

Review Comment:
   Minor: The type this returns should use `StructLike` rather than `Record`. While this will return `Record`, there is no need to require it. And like the Avro readers, we plan to extend this to be able to return custom types for specific structures. That way we can use it to produce things like `ManifestFile`.
   
   I was able to get the types working like this:
   ```java
   public class InternalReader<T extends StructLike> extends BaseParquetReaders<T> {
   
     private static final InternalReader<?> INSTANCE = new InternalReader<>();
   
     private InternalReader() {}
   
     @SuppressWarnings("unchecked")
     public static <T extends StructLike> ParquetValueReader<T> create(Schema expectedSchema, MessageType fileSchema) {
       return (ParquetValueReader<T>) INSTANCE.createReader(expectedSchema, fileSchema);
     }
   
     @SuppressWarnings("unchecked")
     public static <T extends StructLike> ParquetValueReader<T> create(
         Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
       return (ParquetValueReader<T>) INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
     }
   
     @Override
     @SuppressWarnings("unchecked")
     protected ParquetValueReader<T> createStructReader(
         List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
       return (ParquetValueReader<T>) ParquetValueReaders.recordReader(types, fieldReaders, structType);
     }
   
     ...
   }
   ```
   
   We should be able to clean up some of the suppressions once `BaseParquetReader` is package-private.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917295503


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   I think this has to return a generic fixed reader here to match the existing behavior. Because this class is public, people may have created subclasses for other object models, and `iceberg-parquet` is a module that is covered by `revapi` so we make guarantees about API and behavior stability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917497895


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Optional;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.PrimitiveWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> buildWriter(MessageType type) {
+    return INSTANCE.createWriter(type);
+  }
+
+  @Override
+  protected StructWriter<StructLike> createStructWriter(List<ParquetValueWriter<?>> writers) {
+    return new StructLikeWriter(writers);
+  }
+
+  @Override
+  protected PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return ParquetValueWriters.fixedBuffer(desc);
+  }
+
+  @Override
+  protected Optional<PrimitiveWriter<?>> uuidWriter(ColumnDescriptor desc) {
+    return Optional.of(ParquetValueWriters.uuids(desc));
+  }
+
+  private static class StructLikeWriter extends StructWriter<StructLike> {

Review Comment:
   I don't think we need both `RecordWriter` and `StructLikeWriter` because this can be used to write records. Can you remove `RecordWriter` and use this instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917308787


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -92,4 +151,124 @@ protected void set(Record struct, int pos, Object value) {
       struct.set(pos, value);
     }
   }
+
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  private static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {
+    private DateReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDate read(LocalDate reuse) {
+      return EPOCH_DAY.plusDays(column.nextInteger());
+    }
+  }
+
+  private static class TimestampReader extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampMillisReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampInt96Reader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private static final long UNIX_EPOCH_JULIAN = 2_440_588L;
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      final long timeOfDayNanos = byteBuffer.getLong();
+      final int julianDay = byteBuffer.getInt();
+
+      return Instant.ofEpochMilli(TimeUnit.DAYS.toMillis(julianDay - UNIX_EPOCH_JULIAN))
+          .plusNanos(timeOfDayNanos)
+          .atOffset(ZoneOffset.UTC);
+    }
+  }
+
+  private static class TimestamptzReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimestamptzMillisReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimeMillisReader extends ParquetValueReaders.PrimitiveReader<LocalTime> {
+    private TimeMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalTime read(LocalTime reuse) {
+      return LocalTime.ofNanoOfDay(column.nextLong() * 1000000L);

Review Comment:
   Millisecond time must be stored as an INT32 according to the spec, so I think this should call `column.nextLong` instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918338951


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timeReader(ColumnDescriptor desc, TimeUnit unit) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, TimeUnit unit, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {

Review Comment:
   UUID is unsupported by generic so far. I got an error. 
   
   I can keep it local to this file if we don't need factory method. Some has factory method and some are local to class. It is little confusing, I will take a step back and analyze, when it has to be factory and when local.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918338951


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timeReader(ColumnDescriptor desc, TimeUnit unit) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, TimeUnit unit, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {

Review Comment:
   UUID is unsupported by generic so far. I got an error. 
   
   I can keep it local to this file if we don't need factory method. Some has factory method and some are local to class. It is little confusing, I will take a step back and analyze, when it has to be factory and when local.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1919619088


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timeReader(ColumnDescriptor desc, TimeUnit unit) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, TimeUnit unit, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {

Review Comment:
   I see. So the Generic writers and readers never handled UUID !



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1920166221


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    return null;
+  }
+
+  protected Optional<ParquetValueReader<?>> dateReader(ColumnDescriptor desc) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timeReader(ColumnDescriptor desc, TimeUnit unit) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> timestampReader(
+      ColumnDescriptor desc, TimeUnit unit, boolean isAdjustedToUTC) {
+    return Optional.empty();
+  }
+
+  protected Optional<ParquetValueReader<?>> uuidReader(ColumnDescriptor desc) {

Review Comment:
   CI just failed for `ArrowReaderTest > testReadAll()`. 
   
   Generic reader and writer expects byte[] based UUID handling and internal reader and writer works directly on UUID object. So, we do need a factory method here as we need to supply different kinds of UUID representation. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906066082


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Optional;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> buildWriter(MessageType type) {
+    return INSTANCE.createWriter(type);
+  }
+
+  @Override
+  protected StructWriter<StructLike> createStructWriter(List<ParquetValueWriter<?>> writers) {
+    return new ParquetStructWriter(writers);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<
+          ParquetValueWriters.PrimitiveWriter<?>>
+      logicalTypeWriterVisitor(ColumnDescriptor desc) {
+    return new LogicalTypeWriterVisitor(desc);
+  }
+
+  @Override
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    // accepts ByteBuffer and internally writes as binary.
+    return ParquetValueWriters.byteBuffers(desc);
+  }
+
+  private static class ParquetStructWriter extends StructWriter<StructLike> {
+    private ParquetStructWriter(List<ParquetValueWriter<?>> writers) {
+      super(writers);
+    }
+
+    @Override
+    protected Object get(StructLike struct, int index) {
+      return struct.get(index, Object.class);
+    }
+  }
+
+  private static class LogicalTypeWriterVisitor
+      implements LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<

Review Comment:
   I think it would be better to import `LogicalTypeAnnotation.LogicalTypeAnnotationVisitor` and `ParquetValueWriters.PrimitiveWriter` to fix the formatting here. I know it matches what was copied, but that had been auto-formatted when the project moved to Google style.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906140132


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetWriter.java:
##########
@@ -38,6 +50,19 @@ protected StructWriter<Record> createStructWriter(List<ParquetValueWriter<?>> wr
     return new RecordWriter(writers);
   }
 
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<
+          ParquetValueWriters.PrimitiveWriter<?>>
+      logicalTypeWriterVisitor(ColumnDescriptor desc) {
+    return new LogicalTypeWriterVisitor(desc);
+  }
+
+  @Override
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    // accepts byte[] and internally writes as binary.

Review Comment:
   Nit: unhelpful comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925920839


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +80,46 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReader<?> fixedReader(ColumnDescriptor desc) {
+    return new FixedReader(desc);
+  }
+
+  protected ParquetValueReader<?> dateReader(ColumnDescriptor desc) {
+    return new DateReader(desc);
+  }
+
+  protected ParquetValueReader<?> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    switch (unit) {
+      case MICROS:
+        return new TimeReader(desc);
+      case MILLIS:
+        return new TimeMillisReader(desc);
+      default:
+        throw new UnsupportedOperationException("Unsupported Unit: " + unit);
+    }
+  }
+
+  protected ParquetValueReader<?> timestampReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit, boolean isAdjustedToUTC) {
+    switch (unit) {
+      case MICROS:
+        return isAdjustedToUTC ? new TimestamptzReader(desc) : new TimestampReader(desc);
+      case MILLIS:
+        return isAdjustedToUTC
+            ? new TimestamptzMillisReader(desc)
+            : new TimestampMillisReader(desc);
+      case NANOS:
+        if (isAdjustedToUTC) {
+          return new TimestampInt96Reader(desc);

Review Comment:
   I don't think that we need an extra factory method for int96 timestamps. It is a timestamp, it just is a case that uses a different physical type. I like minimizing the number of dependencies between the parent and child class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925905904


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +46,31 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.FixedWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.DateWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.TimeWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timestampWriter(ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new GenericParquetWriter.TimestamptzWriter(desc);
+    } else {
+      return new GenericParquetWriter.TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer (as FIXED_LEN_BYTE_ARRAY); no special writer needed.
+    return null;

Review Comment:
   I think I commented on this in the last round of reviews. This isn't correct. Incoming values are of type `UUID` so this needs a writer that can convert UUID into a byte array. This should return `ParquetValueWriters.uuids(desc)`.
   
   There's also no need to add a method for this because it is the same between the generic and internal object models.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1926371943


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +46,31 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.FixedWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.DateWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new GenericParquetWriter.TimeWriter(desc);
+  }
+
+  protected ParquetValueWriter<?> timestampWriter(ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new GenericParquetWriter.TimestamptzWriter(desc);
+    } else {
+      return new GenericParquetWriter.TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer (as FIXED_LEN_BYTE_ARRAY); no special writer needed.
+    return null;

Review Comment:
   Sorry, the existing testcases was the reason for confusion as I mentioned in https://github.com/apache/iceberg/pull/11904#discussion_r1926274339
   
   I will update the testcases of Arrow too in this PR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1925697904


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -192,13 +222,17 @@ public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.DateLogicalTypeAnnotation dateType) {
-      return Optional.of(new DateWriter(desc));
+      return Optional.ofNullable(dateWriter(desc));
     }
 
     @Override
     public Optional<ParquetValueWriters.PrimitiveWriter<?>> visit(
         LogicalTypeAnnotation.TimeLogicalTypeAnnotation timeType) {
-      return Optional.of(new TimeWriter(desc));
+      Preconditions.checkArgument(
+          LogicalTypeAnnotation.TimeUnit.MICROS.equals(timeType.getUnit()),
+          "Cannot write time in %s, only MICROS is supported",
+          timeType.getUnit());
+      return Optional.ofNullable(timeWriter(desc));

Review Comment:
   Because the `InternalWriter` returns null for `timewriter()` meaning use the primitive writer (no need of special handling). Existing `Optional.of()` will give NPE. To avoid it, I used  `Optional.ofNullable()` so it returns `Optional.empty()` when it is null. 
   
   I will do the null check manually and return super method and revert back to `Optional.of()`, if it looks odd. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1926270438


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueReaders.java:
##########
@@ -850,4 +919,42 @@ private TripleIterator<?> firstNonNullColumn(List<TripleIterator<?>> columns) {
       return NullReader.NULL_COLUMN;
     }
   }
+
+  private static class RecordReader<T extends StructLike> extends StructReader<T, T> {

Review Comment:
   Internal readers expects `ParquetValueReader<StructLike>` and generic reader expects `ParquetValueReader<Record>`, even though the `Record` extends `StructLike`, because of java generics, we need to return the specific type here. We cannot cast `ParquetValueReader<Record>` to `ParquetValueReader<StructLike>`. 
   
   Hence, this was refactored. Similar to the writer refactoring.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906075455


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {

Review Comment:
   Here also, there's not much value in using `Parquet` in the class name. Since this will produce `GenericRecord` instances, how about `RecordReader`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906072738


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {
+    private final GenericRecord template;

Review Comment:
   The basic implementation of `StructLike` is `org.apache.iceberg.data.GenericRecord`. Like the Avro internal readers, this should produce `GenericRecord` instances.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906069631


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalWriter.java:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Optional;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueWriter;
+import org.apache.iceberg.parquet.ParquetValueWriters;
+import org.apache.iceberg.parquet.ParquetValueWriters.StructWriter;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.types.Type;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+
+/**
+ * A Writer that consumes Iceberg's internal in-memory object model.
+ *
+ * <p>Iceberg's internal in-memory object model produces the types defined in {@link
+ * Type.TypeID#javaClass()}.
+ */
+public class InternalWriter extends BaseParquetWriter<StructLike> {
+  private static final InternalWriter INSTANCE = new InternalWriter();
+
+  private InternalWriter() {}
+
+  public static ParquetValueWriter<StructLike> buildWriter(MessageType type) {
+    return INSTANCE.createWriter(type);
+  }
+
+  @Override
+  protected StructWriter<StructLike> createStructWriter(List<ParquetValueWriter<?>> writers) {
+    return new ParquetStructWriter(writers);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<
+          ParquetValueWriters.PrimitiveWriter<?>>
+      logicalTypeWriterVisitor(ColumnDescriptor desc) {
+    return new LogicalTypeWriterVisitor(desc);
+  }
+
+  @Override
+  protected ParquetValueWriters.PrimitiveWriter<?> fixedWriter(ColumnDescriptor desc) {
+    // accepts ByteBuffer and internally writes as binary.

Review Comment:
   I don't think this comment is very helpful. Probably remove it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1908250543


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.parquet.ParquetValueReaders.StructReader;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.PrimitiveType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> buildReader(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetStructReader(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<ParquetValueReader<?>>
+      logicalTypeReaderVisitor(
+          ColumnDescriptor desc,
+          org.apache.iceberg.types.Type.PrimitiveType expected,
+          PrimitiveType primitive) {
+    return new LogicalTypeAnnotationParquetValueReaderVisitor(desc, expected, primitive);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReaders.PrimitiveReader<?> int96Reader(ColumnDescriptor desc) {
+    // normal handling as int96
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  private static class ParquetStructReader extends StructReader<StructLike, StructLike> {

Review Comment:
   Cannot reuse the class from `GenericParquetReaders` as  it is based on `Record` interface, we need a class based on `StructLike` interface. 
   
   I will rename to `StructLikeReader`, just like the `StructLikeWriter` from `InternalWriter` class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#issuecomment-2598582546

   @rdblue: Thanks for the review. I have addressed all the comments. Please take another look.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1906140419


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetWriter.java:
##########
@@ -38,6 +50,19 @@ protected StructWriter<Record> createStructWriter(List<ParquetValueWriter<?>> wr
     return new RecordWriter(writers);
   }
 
+  @Override
+  protected LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<
+          ParquetValueWriters.PrimitiveWriter<?>>
+      logicalTypeWriterVisitor(ColumnDescriptor desc) {
+    return new LogicalTypeWriterVisitor(desc);

Review Comment:
   Here, I would also prefer not to have subclasses provide the visitor.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#issuecomment-2612954116

   rebasing the PR as Flink hit a flaky test https://github.com/apache/iceberg/issues/11833#issuecomment-2581584085


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924377935


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/InternalReader.java:
##########
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.parquet.ParquetValueReader;
+import org.apache.iceberg.parquet.ParquetValueReaders;
+import org.apache.iceberg.types.Types.StructType;
+import org.apache.parquet.column.ColumnDescriptor;
+import org.apache.parquet.schema.LogicalTypeAnnotation;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.Type;
+
+public class InternalReader extends BaseParquetReaders<StructLike> {
+
+  private static final InternalReader INSTANCE = new InternalReader();
+
+  private InternalReader() {}
+
+  public static ParquetValueReader<StructLike> create(
+      Schema expectedSchema, MessageType fileSchema) {
+    return INSTANCE.createReader(expectedSchema, fileSchema);
+  }
+
+  public static ParquetValueReader<StructLike> create(
+      Schema expectedSchema, MessageType fileSchema, Map<Integer, ?> idToConstant) {
+    return INSTANCE.createReader(expectedSchema, fileSchema, idToConstant);
+  }
+
+  @Override
+  protected ParquetValueReader<StructLike> createStructReader(
+      List<Type> types, List<ParquetValueReader<?>> fieldReaders, StructType structType) {
+    return new ParquetValueReaders.RecordReader<>(types, fieldReaders, structType);
+  }
+
+  @Override
+  protected ParquetValueReader<?> fixedReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.BytesReader(desc);
+  }
+
+  @Override
+  protected ParquetValueReader<?> dateReader(ColumnDescriptor desc) {
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  @Override
+  protected ParquetValueReader<?> timeReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return ParquetValueReaders.timestampMillisReader(desc);
+    }
+
+    return new ParquetValueReaders.UnboxedReader<>(desc);
+  }
+
+  @Override
+  protected ParquetValueReader<?> timestampReader(
+      ColumnDescriptor desc, LogicalTypeAnnotation.TimeUnit unit, boolean isAdjustedToUTC) {
+    if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+      return ParquetValueReaders.timestampMillisReader(desc);
+    } else if (unit == LogicalTypeAnnotation.TimeUnit.NANOS) {
+      return ParquetValueReaders.timestampInt96Reader(desc);

Review Comment:
   This should be fixed as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924379496


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetWriter.java:
##########
@@ -50,6 +54,32 @@ protected ParquetValueWriter<T> createWriter(MessageType type) {
   protected abstract ParquetValueWriters.StructWriter<T> createStructWriter(
       List<ParquetValueWriter<?>> writers);
 
+  protected ParquetValueWriter<?> fixedWriter(ColumnDescriptor desc) {
+    return new FixedWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> dateWriter(ColumnDescriptor desc) {
+    return new DateWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timeWriter(ColumnDescriptor desc) {
+    return new TimeWriter(desc);
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> timestampWriter(
+      ColumnDescriptor desc, boolean isAdjustedToUTC) {
+    if (isAdjustedToUTC) {
+      return new TimestamptzWriter(desc);
+    } else {
+      return new TimestampWriter(desc);
+    }
+  }
+
+  protected ParquetValueWriters.PrimitiveWriter<?> uuidWriter(ColumnDescriptor desc) {
+    // Use primitive-type writer; no special writer needed.

Review Comment:
   There is no primitive writer for UUIDs. Maybe this was a copy/paste error?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1924391656


##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueWriters.java:
##########
@@ -330,6 +361,37 @@ public void write(int repetitionLevel, CharSequence value) {
     }
   }
 
+  private static class UUIDWriter extends PrimitiveWriter<UUID> {
+    private static final ThreadLocal<ByteBuffer> BUFFER =
+        ThreadLocal.withInitial(
+            () -> {
+              ByteBuffer buffer = ByteBuffer.allocate(16);
+              buffer.order(ByteOrder.BIG_ENDIAN);
+              return buffer;
+            });
+
+    private UUIDWriter(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public void write(int repetitionLevel, UUID value) {
+      ByteBuffer buffer = UUIDUtil.convertToByteBuffer(value, BUFFER.get());
+      column.writeBinary(repetitionLevel, Binary.fromReusedByteBuffer(buffer));
+    }
+  }
+
+  public static class RecordWriter<T extends StructLike> extends StructWriter<T> {

Review Comment:
   Rather than adding a public class, can you add a factory method for this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917264392


##########
api/src/test/java/org/apache/iceberg/util/RandomUtil.java:
##########
@@ -246,15 +246,15 @@ public static List<Object> generateList(
       if (list.isElementOptional() && random.nextInt(20) == 1) {
         result.add(null);
       } else {
-        result.add(elementResult.get());
+        result.add(elements.get());
       }
     }
 
     return result;
   }
 
   public static Map<Object, Object> generateMap(
-      Random random, Types.MapType map, Supplier<Object> keyResult, Supplier<Object> valueResult) {
+      Random random, Types.MapType map, Supplier<Object> keyResult, Supplier<Object> values) {

Review Comment:
   Similarly, I think `valueSupplier` is a better name. This also doesn't address the problem for `keyResult`, which should also be `keySupplier`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917266691


##########
data/src/test/java/org/apache/iceberg/data/parquet/TestInternalData.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.InternalTestHelpers;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RandomInternalData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.DataTest;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+public class TestInternalData extends DataTest {
+  @Override
+  protected void writeAndValidate(Schema schema) throws IOException {
+    writeAndValidate(schema, schema);
+  }
+
+  @Override
+  protected void writeAndValidate(Schema writeSchema, Schema expectedSchema) throws IOException {
+    List<StructLike> expected = RandomInternalData.generate(writeSchema, 100, 42L);

Review Comment:
   As I mentioned on the Avro one, it would be better to use a new random seed for each test.



##########
data/src/test/java/org/apache/iceberg/data/parquet/TestInternalData.java:
##########
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data.parquet;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+import org.apache.iceberg.Files;
+import org.apache.iceberg.InternalTestHelpers;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.RandomInternalData;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.data.DataTest;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.parquet.Parquet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+
+public class TestInternalData extends DataTest {
+  @Override
+  protected void writeAndValidate(Schema schema) throws IOException {
+    writeAndValidate(schema, schema);
+  }
+
+  @Override
+  protected void writeAndValidate(Schema writeSchema, Schema expectedSchema) throws IOException {
+    List<StructLike> expected = RandomInternalData.generate(writeSchema, 100, 42L);

Review Comment:
   As I mentioned on the Avro one, it would be better to use a new random seed for this test suite.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917286696


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   I think this still needs to return the generic fixed reader. Otherwise there is a behavior change. It's too bad that this class is public.



##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   I think this still needs to return the generic fixed reader. Otherwise there is a behavior change. It's too bad that this class is public.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917285311


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -397,7 +404,7 @@ public ParquetValueReader<?> primitive(
         case INT96:
           // Impala & Spark used to write timestamps as INT96 without a logical type. For backwards
           // compatibility we try to read INT96 as timestamps.
-          return new TimestampInt96Reader(desc);
+          return int96Reader(desc);

Review Comment:
   Can't this call `timestampReader(desc, NANOS, ((TimestampType) primitive).isAdjustedToUTC())` instead? The primitive type is available from `desc` so that method can check and we don't need to have a factory method for INT96.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["rdblue (via GitHub)" <gi...@apache.org>]

rdblue commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1917305124


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -76,6 +67,31 @@ protected ParquetValueReader<T> createReader(
   protected abstract ParquetValueReader<T> createStructReader(
       List<Type> types, List<ParquetValueReader<?>> fieldReaders, Types.StructType structType);
 
+  protected ParquetValueReaders.PrimitiveReader<?> fixedReader(ColumnDescriptor desc) {
+    return null;

Review Comment:
   Since this class is not widely used, could you also mark it deprecated so that we can make it package-private in 1.9.0?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918335337


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/BaseParquetReaders.java:
##########
@@ -397,7 +404,7 @@ public ParquetValueReader<?> primitive(
         case INT96:
           // Impala & Spark used to write timestamps as INT96 without a logical type. For backwards
           // compatibility we try to read INT96 as timestamps.
-          return new TimestampInt96Reader(desc);
+          return int96Reader(desc);

Review Comment:
   We don't have a nano handling in `timestampReader` yet. So, you are suggesting to rename `TimestampInt96Reader` to `TimestampNanoReader` and call it from `timestampReader`? So, it can also be used for nano reading in future? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] Parquet: Add readers and writers for the internal object model [iceberg]

["ajantha-bhat (via GitHub)" <gi...@apache.org>]

ajantha-bhat commented on code in PR #11904:
URL: https://github.com/apache/iceberg/pull/11904#discussion_r1918344913


##########
parquet/src/main/java/org/apache/iceberg/data/parquet/GenericParquetReaders.java:
##########
@@ -92,4 +151,124 @@ protected void set(Record struct, int pos, Object value) {
       struct.set(pos, value);
     }
   }
+
+  private static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC);
+  private static final LocalDate EPOCH_DAY = EPOCH.toLocalDate();
+
+  private static class DateReader extends ParquetValueReaders.PrimitiveReader<LocalDate> {
+    private DateReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDate read(LocalDate reuse) {
+      return EPOCH_DAY.plusDays(column.nextInteger());
+    }
+  }
+
+  private static class TimestampReader extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampMillisReader
+      extends ParquetValueReaders.PrimitiveReader<LocalDateTime> {
+    private TimestampMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalDateTime read(LocalDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS).toLocalDateTime();
+    }
+  }
+
+  private static class TimestampInt96Reader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private static final long UNIX_EPOCH_JULIAN = 2_440_588L;
+
+    private TimestampInt96Reader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      final ByteBuffer byteBuffer =
+          column.nextBinary().toByteBuffer().order(ByteOrder.LITTLE_ENDIAN);
+      final long timeOfDayNanos = byteBuffer.getLong();
+      final int julianDay = byteBuffer.getInt();
+
+      return Instant.ofEpochMilli(TimeUnit.DAYS.toMillis(julianDay - UNIX_EPOCH_JULIAN))
+          .plusNanos(timeOfDayNanos)
+          .atOffset(ZoneOffset.UTC);
+    }
+  }
+
+  private static class TimestamptzReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong(), ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimestamptzMillisReader
+      extends ParquetValueReaders.PrimitiveReader<OffsetDateTime> {
+    private TimestamptzMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public OffsetDateTime read(OffsetDateTime reuse) {
+      return EPOCH.plus(column.nextLong() * 1000, ChronoUnit.MICROS);
+    }
+  }
+
+  private static class TimeMillisReader extends ParquetValueReaders.PrimitiveReader<LocalTime> {
+    private TimeMillisReader(ColumnDescriptor desc) {
+      super(desc);
+    }
+
+    @Override
+    public LocalTime read(LocalTime reuse) {
+      return LocalTime.ofNanoOfDay(column.nextLong() * 1000000L);

Review Comment:
   how do I test these? `DataTest` doesn't handle time units and writers write in micro always? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org