[GitHub] [iceberg] chenjunjiedada commented on a change in pull request #1129: Turn on name mapping for avro

[GitBox <gi...@apache.org>]

chenjunjiedada commented on a change in pull request #1129:
URL: https://github.com/apache/iceberg/pull/1129#discussion_r443989238



##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestSparkTableUtil.java
##########
@@ -298,4 +309,69 @@ public void testImportWithNameMappingForVectorizedParquetReader() throws Excepti
 
     Assert.assertEquals(expected.stream().map(SimpleRecord::getData).collect(Collectors.toList()), actual);
   }
+
+  @Test
+  public void testAvroReaderWithNameMapping() throws IOException {

Review comment:
       will do.

##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestSparkTableUtil.java
##########
@@ -298,4 +309,69 @@ public void testImportWithNameMappingForVectorizedParquetReader() throws Excepti
 
     Assert.assertEquals(expected.stream().map(SimpleRecord::getData).collect(Collectors.toList()), actual);
   }
+
+  @Test
+  public void testAvroReaderWithNameMapping() throws IOException {
+    File avroFile = temp.newFile();
+    org.apache.avro.Schema avroSchema = SchemaBuilder.record("TestRecord")
+        .namespace("org.apache.iceberg.spark.data")
+        .fields()
+        .requiredInt("id")
+        .requiredString("name")
+        .endRecord();
+
+    GenericRecord record1 = new GenericData.Record(avroSchema);
+    record1.put("id", 1);
+    record1.put("name", "Bob");
+
+    GenericRecord record2 = new GenericData.Record(avroSchema);
+    record2.put("id", 2);
+    record2.put("name", "Alice");
+
+    DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(avroSchema);
+    DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter);
+
+    dataFileWriter.create(avroSchema, avroFile);
+    dataFileWriter.append(record1);
+    dataFileWriter.append(record2);
+    dataFileWriter.close();
+
+    DataFile avroDataFile = DataFiles.builder(PartitionSpec.unpartitioned())
+        .withFormat("avro")
+        .withFileSizeInBytes(avroFile.length())
+        .withPath(avroFile.getAbsolutePath())
+        .withRecordCount(2)
+        .build();
+
+    Schema filteredSchema = new Schema(
+        required(1, "name", Types.StringType.get())
+    );
+
+    NameMapping nameMapping = MappingUtil.create(filteredSchema);

Review comment:
       We can not use a full schema for the table because it cannot build Avro read schema (with `name` and `id` column) by using pruned schema (with `name` column) in `ProjectionDatumReader#setSchema`.
   
   Also, `setSchema` will ignore the name mapping if the Avro file schema has Ids. So we might need to utilize the `RemoveIds` from the Avro test via moving it to source dir and changing it to the public class. What do you think about this?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org