You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "anoopj (via GitHub)" <gi...@apache.org> on 2025/07/29 19:00:56 UTC

[PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

anoopj opened a new pull request, #13699:
URL: https://github.com/apache/iceberg/pull/13699

   Pure refactoring change. Also added a couple of unit tests for nested structs/lists.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#issuecomment-3197388739

   I like the PR. We need a bit more coverage for the complex cases, and then I think we are fine


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2248978982


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,133 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));

Review Comment:
   Rather than operate field-by-field like this, it would make more sense for the schema visitor to take the passed in schema and convert to perform the complete conversion that way.  This approach is mixing the patterns, but the visitor pattern covers this as well.
   
   That would make this a straightforward:
   
   ```java
   return TypeUtil.visit(schema, new IcebertToArrowTypeConvertor(schema));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2283621660


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,134 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {

Review Comment:
   nit: Why not call `TypeUtil.visit` on the schema directly like other visitors? it essentially pushes the for loop inside the visitor, which is needed for nested struct anyway.
   
   e.g. Spark `SparkSchemaUtil`
   ```
     public static StructType convert(Schema schema) {
       return (StructType) TypeUtil.visit(schema, new TypeToSparkType());
     }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2249002781


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,133 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));
     }
-
     return new Schema(fields.build());
   }
 
   public static Field convert(final NestedField field) {
-    final ArrowType arrowType;
-
-    final List<Field> children = Lists.newArrayList();
-    Map<String, String> metadata = null;
-
-    switch (field.type().typeId()) {
-      case BINARY:
-        arrowType = ArrowType.Binary.INSTANCE;
-        break;
-      case FIXED:
-        final Types.FixedType fixedType = (Types.FixedType) field.type();
-        arrowType = new ArrowType.FixedSizeBinary(fixedType.length());
-        break;
-      case BOOLEAN:
-        arrowType = ArrowType.Bool.INSTANCE;
-        break;
-      case INTEGER:
-        arrowType = new ArrowType.Int(Integer.SIZE, true /* signed */);
-        break;
-      case LONG:
-        arrowType = new ArrowType.Int(Long.SIZE, true /* signed */);
-        break;
-      case FLOAT:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
-        break;
-      case DOUBLE:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
-        break;
-      case DECIMAL:
-        final Types.DecimalType decimalType = (Types.DecimalType) field.type();
-        arrowType = new ArrowType.Decimal(decimalType.precision(), decimalType.scale(), 128);
-        break;
-      case STRING:
-        arrowType = ArrowType.Utf8.INSTANCE;
-        break;
-      case TIME:
-        arrowType = new ArrowType.Time(TimeUnit.MICROSECOND, Long.SIZE);
-        break;
-      case UUID:
-        arrowType = new ArrowType.FixedSizeBinary(16);
-        break;
-      case TIMESTAMP:
-        arrowType =
-            new ArrowType.Timestamp(
-                TimeUnit.MICROSECOND,
-                ((Types.TimestampType) field.type()).shouldAdjustToUTC() ? "UTC" : null);
-        break;
-      case DATE:
-        arrowType = new ArrowType.Date(DateUnit.DAY);
-        break;
-      case STRUCT:
-        final StructType struct = field.type().asStructType();
-        arrowType = ArrowType.Struct.INSTANCE;
-
-        for (NestedField nested : struct.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case LIST:
-        final ListType listType = field.type().asListType();
-        arrowType = ArrowType.List.INSTANCE;
-
-        for (NestedField nested : listType.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case MAP:
-        metadata = ImmutableMap.of(ORIGINAL_TYPE, MAP_TYPE);
-        final MapType mapType = field.type().asMapType();
-        arrowType = new ArrowType.Map(false);
-        List<Field> entryFields = Lists.transform(mapType.fields(), ArrowSchemaUtil::convert);
-        Field entry =
-            new Field("", new FieldType(field.isOptional(), arrowType, null), entryFields);
-        children.add(entry);
-        break;
-      default:
-        throw new UnsupportedOperationException("Unsupported field type: " + field);
+    return TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field));
+  }
+
+  private static class IcebergToArrowTypeConverter extends TypeUtil.SchemaVisitor<Field> {
+    private final NestedField currentField;
+
+    IcebergToArrowTypeConverter(NestedField field) {
+      this.currentField = field;
+    }
+
+    @Override
+    public Field schema(org.apache.iceberg.Schema schema, Field structResult) {
+      return structResult;
+    }
+
+    @Override
+    public Field struct(StructType struct, List<Field> fieldResults) {
+      ArrowType arrowType = ArrowType.Struct.INSTANCE;

Review Comment:
   I don't think we need to assign the type to a variable like this.  It can be inlined.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2249015552


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,133 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));
     }
-
     return new Schema(fields.build());
   }
 
   public static Field convert(final NestedField field) {
-    final ArrowType arrowType;
-
-    final List<Field> children = Lists.newArrayList();
-    Map<String, String> metadata = null;
-
-    switch (field.type().typeId()) {
-      case BINARY:
-        arrowType = ArrowType.Binary.INSTANCE;
-        break;
-      case FIXED:
-        final Types.FixedType fixedType = (Types.FixedType) field.type();
-        arrowType = new ArrowType.FixedSizeBinary(fixedType.length());
-        break;
-      case BOOLEAN:
-        arrowType = ArrowType.Bool.INSTANCE;
-        break;
-      case INTEGER:
-        arrowType = new ArrowType.Int(Integer.SIZE, true /* signed */);
-        break;
-      case LONG:
-        arrowType = new ArrowType.Int(Long.SIZE, true /* signed */);
-        break;
-      case FLOAT:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
-        break;
-      case DOUBLE:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
-        break;
-      case DECIMAL:
-        final Types.DecimalType decimalType = (Types.DecimalType) field.type();
-        arrowType = new ArrowType.Decimal(decimalType.precision(), decimalType.scale(), 128);
-        break;
-      case STRING:
-        arrowType = ArrowType.Utf8.INSTANCE;
-        break;
-      case TIME:
-        arrowType = new ArrowType.Time(TimeUnit.MICROSECOND, Long.SIZE);
-        break;
-      case UUID:
-        arrowType = new ArrowType.FixedSizeBinary(16);
-        break;
-      case TIMESTAMP:
-        arrowType =
-            new ArrowType.Timestamp(
-                TimeUnit.MICROSECOND,
-                ((Types.TimestampType) field.type()).shouldAdjustToUTC() ? "UTC" : null);
-        break;
-      case DATE:
-        arrowType = new ArrowType.Date(DateUnit.DAY);
-        break;
-      case STRUCT:
-        final StructType struct = field.type().asStructType();
-        arrowType = ArrowType.Struct.INSTANCE;
-
-        for (NestedField nested : struct.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case LIST:
-        final ListType listType = field.type().asListType();
-        arrowType = ArrowType.List.INSTANCE;
-
-        for (NestedField nested : listType.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case MAP:
-        metadata = ImmutableMap.of(ORIGINAL_TYPE, MAP_TYPE);
-        final MapType mapType = field.type().asMapType();
-        arrowType = new ArrowType.Map(false);
-        List<Field> entryFields = Lists.transform(mapType.fields(), ArrowSchemaUtil::convert);
-        Field entry =
-            new Field("", new FieldType(field.isOptional(), arrowType, null), entryFields);
-        children.add(entry);
-        break;
-      default:
-        throw new UnsupportedOperationException("Unsupported field type: " + field);
+    return TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field));
+  }
+
+  private static class IcebergToArrowTypeConverter extends TypeUtil.SchemaVisitor<Field> {
+    private final NestedField currentField;
+
+    IcebergToArrowTypeConverter(NestedField field) {
+      this.currentField = field;
+    }
+
+    @Override
+    public Field schema(org.apache.iceberg.Schema schema, Field structResult) {
+      return structResult;
+    }
+
+    @Override
+    public Field struct(StructType struct, List<Field> fieldResults) {
+      ArrowType arrowType = ArrowType.Struct.INSTANCE;

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2283640724


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,134 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {

Review Comment:
   @stevenzwu I had the [same comment here](https://github.com/apache/iceberg/pull/13699#discussion_r2248978982), but the issue is around Arrow types and how they handle the enclosing schema (it's not a struct, just a list of fields).  It becomes awkward to convert at the schema level because it would require the generic type be the arrow `Schema` which is only constructed after visiting the full schema.  Being able to return `Field` is more consistent with the behavior we want out of visiting and translating the type.
   
   This is also consistent with patterns we have with converting schema's to parquet ([example here](https://github.com/apache/iceberg/blob/018710edd56652e915fe858cba512b3064efcfa9/parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java#L81))



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2283645644


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,134 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {

Review Comment:
   looks like this is due to the behavior of Arrow `Schema`. we can go with that model if it is `TypeToArrowType`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2282735828


##########
arrow/src/test/java/org/apache/iceberg/arrow/TestArrowSchemaUtil.java:
##########
@@ -101,6 +102,53 @@ public void convertComplex() {
     assertThat(arrow.getFields()).hasSameSizeAs(iceberg.columns());

Review Comment:
   Could we add a bit more assertions here for the map type?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2249170947


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,131 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));
     }
-

Review Comment:
   Nit: why remove this line?
   The style guide requires a newline after every block



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2249171834


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,131 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));
     }
-
     return new Schema(fields.build());
   }
 
   public static Field convert(final NestedField field) {
-    final ArrowType arrowType;
-
-    final List<Field> children = Lists.newArrayList();
-    Map<String, String> metadata = null;
-
-    switch (field.type().typeId()) {
-      case BINARY:
-        arrowType = ArrowType.Binary.INSTANCE;
-        break;
-      case FIXED:
-        final Types.FixedType fixedType = (Types.FixedType) field.type();
-        arrowType = new ArrowType.FixedSizeBinary(fixedType.length());
-        break;
-      case BOOLEAN:
-        arrowType = ArrowType.Bool.INSTANCE;
-        break;
-      case INTEGER:
-        arrowType = new ArrowType.Int(Integer.SIZE, true /* signed */);
-        break;
-      case LONG:
-        arrowType = new ArrowType.Int(Long.SIZE, true /* signed */);
-        break;
-      case FLOAT:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
-        break;
-      case DOUBLE:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
-        break;
-      case DECIMAL:
-        final Types.DecimalType decimalType = (Types.DecimalType) field.type();
-        arrowType = new ArrowType.Decimal(decimalType.precision(), decimalType.scale(), 128);
-        break;
-      case STRING:
-        arrowType = ArrowType.Utf8.INSTANCE;
-        break;
-      case TIME:
-        arrowType = new ArrowType.Time(TimeUnit.MICROSECOND, Long.SIZE);
-        break;
-      case UUID:
-        arrowType = new ArrowType.FixedSizeBinary(16);
-        break;
-      case TIMESTAMP:
-        arrowType =
-            new ArrowType.Timestamp(
-                TimeUnit.MICROSECOND,
-                ((Types.TimestampType) field.type()).shouldAdjustToUTC() ? "UTC" : null);
-        break;
-      case DATE:
-        arrowType = new ArrowType.Date(DateUnit.DAY);
-        break;
-      case STRUCT:
-        final StructType struct = field.type().asStructType();
-        arrowType = ArrowType.Struct.INSTANCE;
-
-        for (NestedField nested : struct.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case LIST:
-        final ListType listType = field.type().asListType();
-        arrowType = ArrowType.List.INSTANCE;
-
-        for (NestedField nested : listType.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case MAP:
-        metadata = ImmutableMap.of(ORIGINAL_TYPE, MAP_TYPE);
-        final MapType mapType = field.type().asMapType();
-        arrowType = new ArrowType.Map(false);
-        List<Field> entryFields = Lists.transform(mapType.fields(), ArrowSchemaUtil::convert);
-        Field entry =
-            new Field("", new FieldType(field.isOptional(), arrowType, null), entryFields);
-        children.add(entry);
-        break;
-      default:
-        throw new UnsupportedOperationException("Unsupported field type: " + field);
+    return TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field));
+  }
+
+  private static class IcebergToArrowTypeConverter extends TypeUtil.SchemaVisitor<Field> {
+    private final NestedField currentField;
+
+    IcebergToArrowTypeConverter(NestedField field) {
+      this.currentField = field;
+    }
+
+    @Override
+    public Field schema(org.apache.iceberg.Schema schema, Field structResult) {
+      return structResult;
+    }
+
+    @Override
+    public Field struct(StructType struct, List<Field> fieldResults) {
+      return new Field(
+          currentField.name(),
+          new FieldType(currentField.isOptional(), ArrowType.Struct.INSTANCE, null),
+          convertChildren(struct.fields()));
+    }
+
+    @Override
+    public Field field(NestedField field, Field fieldResult) {
+      return fieldResult;
+    }
+
+    @Override
+    public Field list(ListType list, Field elementResult) {
+      return new Field(
+          currentField.name(),
+          new FieldType(currentField.isOptional(), ArrowType.List.INSTANCE, null),
+          convertChildren(list.fields()));
+    }
+
+    @Override
+    public Field map(MapType map, Field keyResult, Field valueResult) {
+      Map<String, String> metadata = ImmutableMap.of(ORIGINAL_TYPE, MAP_TYPE);
+      ArrowType arrowType = new ArrowType.Map(false);
+
+      List<Field> entryFields = convertChildren(map.fields());
+
+      Field entry =
+          new Field("", new FieldType(currentField.isOptional(), arrowType, null), entryFields);
+      List<Field> children = Lists.newArrayList(entry);
+
+      return new Field(
+          currentField.name(),
+          new FieldType(currentField.isOptional(), arrowType, null, metadata),
+          children);
     }
 
-    return new Field(
-        field.name(), new FieldType(field.isOptional(), arrowType, null, metadata), children);
+    private List<Field> convertChildren(Collection<NestedField> children) {
+      List<Field> converted = Lists.newArrayListWithCapacity(children.size());
+      for (NestedField child : children) {
+        converted.add(TypeUtil.visit(child.type(), new IcebergToArrowTypeConverter(child)));
+      }

Review Comment:
   Nit; newlnie



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2249002337


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,133 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));

Review Comment:
   I think I remember now that arrow doesn't have a top-level structure that you can build incrementally.  This actually looks similar to how we do some of the avro conversion, so I think this is probably fine to keep.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#issuecomment-3201488013

   thanks @anoopj for the contribution and @pvary @danielcweeks for the review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2249015665


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,133 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));

Review Comment:
   Sounds good. Will keep it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu merged PR #13699:
URL: https://github.com/apache/iceberg/pull/13699


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2284062658


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,134 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {

Review Comment:
   didn't see Dan's comment in my follow-up comment. will resolve this thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2250018633


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,131 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));
     }
-

Review Comment:
   Done, but I thought we used the Google Java style, which [doesn't](https://google.github.io/styleguide/javaguide.html#s4.6-whitespace) **require** a new line after every block. 



##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,131 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));
     }
-
     return new Schema(fields.build());
   }
 
   public static Field convert(final NestedField field) {
-    final ArrowType arrowType;
-
-    final List<Field> children = Lists.newArrayList();
-    Map<String, String> metadata = null;
-
-    switch (field.type().typeId()) {
-      case BINARY:
-        arrowType = ArrowType.Binary.INSTANCE;
-        break;
-      case FIXED:
-        final Types.FixedType fixedType = (Types.FixedType) field.type();
-        arrowType = new ArrowType.FixedSizeBinary(fixedType.length());
-        break;
-      case BOOLEAN:
-        arrowType = ArrowType.Bool.INSTANCE;
-        break;
-      case INTEGER:
-        arrowType = new ArrowType.Int(Integer.SIZE, true /* signed */);
-        break;
-      case LONG:
-        arrowType = new ArrowType.Int(Long.SIZE, true /* signed */);
-        break;
-      case FLOAT:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.SINGLE);
-        break;
-      case DOUBLE:
-        arrowType = new ArrowType.FloatingPoint(FloatingPointPrecision.DOUBLE);
-        break;
-      case DECIMAL:
-        final Types.DecimalType decimalType = (Types.DecimalType) field.type();
-        arrowType = new ArrowType.Decimal(decimalType.precision(), decimalType.scale(), 128);
-        break;
-      case STRING:
-        arrowType = ArrowType.Utf8.INSTANCE;
-        break;
-      case TIME:
-        arrowType = new ArrowType.Time(TimeUnit.MICROSECOND, Long.SIZE);
-        break;
-      case UUID:
-        arrowType = new ArrowType.FixedSizeBinary(16);
-        break;
-      case TIMESTAMP:
-        arrowType =
-            new ArrowType.Timestamp(
-                TimeUnit.MICROSECOND,
-                ((Types.TimestampType) field.type()).shouldAdjustToUTC() ? "UTC" : null);
-        break;
-      case DATE:
-        arrowType = new ArrowType.Date(DateUnit.DAY);
-        break;
-      case STRUCT:
-        final StructType struct = field.type().asStructType();
-        arrowType = ArrowType.Struct.INSTANCE;
-
-        for (NestedField nested : struct.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case LIST:
-        final ListType listType = field.type().asListType();
-        arrowType = ArrowType.List.INSTANCE;
-
-        for (NestedField nested : listType.fields()) {
-          children.add(convert(nested));
-        }
-        break;
-      case MAP:
-        metadata = ImmutableMap.of(ORIGINAL_TYPE, MAP_TYPE);
-        final MapType mapType = field.type().asMapType();
-        arrowType = new ArrowType.Map(false);
-        List<Field> entryFields = Lists.transform(mapType.fields(), ArrowSchemaUtil::convert);
-        Field entry =
-            new Field("", new FieldType(field.isOptional(), arrowType, null), entryFields);
-        children.add(entry);
-        break;
-      default:
-        throw new UnsupportedOperationException("Unsupported field type: " + field);
+    return TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field));
+  }
+
+  private static class IcebergToArrowTypeConverter extends TypeUtil.SchemaVisitor<Field> {
+    private final NestedField currentField;
+
+    IcebergToArrowTypeConverter(NestedField field) {
+      this.currentField = field;
+    }
+
+    @Override
+    public Field schema(org.apache.iceberg.Schema schema, Field structResult) {
+      return structResult;
+    }
+
+    @Override
+    public Field struct(StructType struct, List<Field> fieldResults) {
+      return new Field(
+          currentField.name(),
+          new FieldType(currentField.isOptional(), ArrowType.Struct.INSTANCE, null),
+          convertChildren(struct.fields()));
+    }
+
+    @Override
+    public Field field(NestedField field, Field fieldResult) {
+      return fieldResult;
+    }
+
+    @Override
+    public Field list(ListType list, Field elementResult) {
+      return new Field(
+          currentField.name(),
+          new FieldType(currentField.isOptional(), ArrowType.List.INSTANCE, null),
+          convertChildren(list.fields()));
+    }
+
+    @Override
+    public Field map(MapType map, Field keyResult, Field valueResult) {
+      Map<String, String> metadata = ImmutableMap.of(ORIGINAL_TYPE, MAP_TYPE);
+      ArrowType arrowType = new ArrowType.Map(false);
+
+      List<Field> entryFields = convertChildren(map.fields());
+
+      Field entry =
+          new Field("", new FieldType(currentField.isOptional(), arrowType, null), entryFields);
+      List<Field> children = Lists.newArrayList(entry);
+
+      return new Field(
+          currentField.name(),
+          new FieldType(currentField.isOptional(), arrowType, null, metadata),
+          children);
     }
 
-    return new Field(
-        field.name(), new FieldType(field.isOptional(), arrowType, null, metadata), children);
+    private List<Field> convertChildren(Collection<NestedField> children) {
+      List<Field> converted = Lists.newArrayListWithCapacity(children.size());
+      for (NestedField child : children) {
+        converted.add(TypeUtil.visit(child.type(), new IcebergToArrowTypeConverter(child)));
+      }

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "anoopj (via GitHub)" <gi...@apache.org>.
anoopj commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2283262559


##########
arrow/src/test/java/org/apache/iceberg/arrow/TestArrowSchemaUtil.java:
##########
@@ -101,6 +102,53 @@ public void convertComplex() {
     assertThat(arrow.getFields()).hasSameSizeAs(iceberg.columns());

Review Comment:
   Great suggestion. Done. Also renamed the test case to be consistent with the new tests I added for structs/lists.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2283621660


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,134 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {

Review Comment:
   nit: Why not call `TypeUtil.visit` on the schema directly like other visitors? it essentially pushes the for loop inside the visitor, which is needed for nested struct anyway.
   
   e.g. Spark
   ```
     public static StructType convert(Schema schema) {
       return (StructType) TypeUtil.visit(schema, new TypeToSparkType());
     }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Rewrite the Iceberg Arrow schema translation to use the visitor pattern [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #13699:
URL: https://github.com/apache/iceberg/pull/13699#discussion_r2251193758


##########
arrow/src/main/java/org/apache/iceberg/arrow/ArrowSchemaUtil.java:
##########
@@ -51,94 +54,131 @@ private ArrowSchemaUtil() {}
   public static Schema convert(final org.apache.iceberg.Schema schema) {
     ImmutableList.Builder<Field> fields = ImmutableList.builder();
 
-    for (NestedField f : schema.columns()) {
-      fields.add(convert(f));
+    for (NestedField field : schema.columns()) {
+      fields.add(TypeUtil.visit(field.type(), new IcebergToArrowTypeConverter(field)));
     }
-

Review Comment:
   This is Iceberg specific: https://iceberg.apache.org/contribute/#block-spacing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org