You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/04/18 02:55:31 UTC
[GitHub] [iceberg] ConeyLiu opened a new pull request, #4577: Fixes read metadata table failed due to illegal character
ConeyLiu opened a new pull request, #4577:
URL: https://github.com/apache/iceberg/pull/4577
As #4576 described, we got a `SchemaParseException ` when reading the metadata table. The root cause is that Avro does not support a name containing dot(.). For example, `Identity(data.id)`. In this case, the partition field named with `data.id` is illegal for Avro, so we need to rewrite it to meet the Avro field name requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1008682913
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -107,13 +107,15 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String fieldName = AvroSchemaUtil.makeCompatibleName(field.name());
Schema.Field newField =
new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())),
null,
JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ newField.addProp(AvroSchemaUtil.ICEBERG_FIELD_NAME_PROP, field.name());
Review Comment:
> Nit: Can we use fieldName here? (Actually prefer to rename to 'oldFieldName' if you don't mind.
This aligns the code in https://github.com/apache/iceberg/blob/dbb8a404f6632a55acb36e949f0e7b84b643cede/core/src/main/java/org/apache/iceberg/avro/TypeToSchema.java#L110
> Also , do you know the significance of this property?
Actually, I don't know the true meaning. It seems the property is set, but could not find any usage. Maybe for debugging or similar.
> Looks like in the other case its only set if the field name has been changed, should we do that?
I do this in previously commit: https://github.com/apache/iceberg/pull/4577/commits/d54148421d8740f6e792d1567e5c7fe610c12a48. And changed by some comments and advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1013696351
##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java:
##########
@@ -978,6 +1091,32 @@ private Set<String> expectedManifestListPaths(Iterable<Snapshot> snapshots, Long
.collect(Collectors.toSet());
}
+ private void createTableAndAddPartitionOnNestedFiled() throws IOException {
Review Comment:
Split into a dedicated class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1014640565
##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+
+import java.io.File;
+import java.io.IOException;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Streams;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestMetadataTableScansWithPartitionEvolution extends MetadataTableScanTestBase {
+ public TestMetadataTableScansWithPartitionEvolution(int formatVersion) {
+ super(formatVersion);
+ }
+
+ @Before
+ public void createTable() throws IOException {
+ TestTables.clearTables();
+ this.tableDir = temp.newFolder();
+ tableDir.delete();
+
+ Schema schema =
+ new Schema(
+ required(1, "id", Types.IntegerType.get()),
+ required(
+ 2,
+ "nested",
+ Types.StructType.of(Types.NestedField.required(3, "id", Types.IntegerType.get()))));
+ this.metadataDir = new File(tableDir, "metadata");
+ PartitionSpec spec = PartitionSpec.builderFor(schema).identity("id").build();
+
+ this.table = create(schema, spec);
+ table.newFastAppend().appendFile(newDataFile("id=0")).appendFile(newDataFile("id=1")).commit();
+
+ table.updateSpec().addField("nested.id").commit();
+ table
+ .newFastAppend()
+ .appendFile(newDataFile("id=2/nested.id=2"))
+ .appendFile(newDataFile("id=3/nested.id=3"))
+ .commit();
+ }
+
+ @Test
+ public void testManifestsTableWithAddPartitionOnNestedField() throws IOException {
+ Table manifestsTable = new ManifestsTable(table.ops(), table);
+ TableScan scan = manifestsTable.newScan();
+
+ try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
+ Assert.assertEquals("Should have one tasks", 1, Iterables.size(tasks));
Review Comment:
Updated to use assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851852032
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Review Comment:
Also, I think isValidFieldName can be inline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#issuecomment-1101053954
Thanks, @chenjunjiedada for the review. And cc @rdblue @szehon-ho @aokolnychyi @RussellSpitzer, could you help to review this? Thanks in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1013695871
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -107,13 +107,15 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String fieldName = AvroSchemaUtil.makeCompatibleName(field.name());
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
nastra commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1014180932
##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+
+import java.io.File;
+import java.io.IOException;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterators;
+import org.apache.iceberg.relocated.com.google.common.collect.Streams;
+import org.apache.iceberg.types.Types;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestMetadataTableScansWithPartitionEvolution extends MetadataTableScanTestBase {
+ public TestMetadataTableScansWithPartitionEvolution(int formatVersion) {
+ super(formatVersion);
+ }
+
+ @Before
+ public void createTable() throws IOException {
+ TestTables.clearTables();
+ this.tableDir = temp.newFolder();
+ tableDir.delete();
+
+ Schema schema =
+ new Schema(
+ required(1, "id", Types.IntegerType.get()),
+ required(
+ 2,
+ "nested",
+ Types.StructType.of(Types.NestedField.required(3, "id", Types.IntegerType.get()))));
+ this.metadataDir = new File(tableDir, "metadata");
+ PartitionSpec spec = PartitionSpec.builderFor(schema).identity("id").build();
+
+ this.table = create(schema, spec);
+ table.newFastAppend().appendFile(newDataFile("id=0")).appendFile(newDataFile("id=1")).commit();
+
+ table.updateSpec().addField("nested.id").commit();
+ table
+ .newFastAppend()
+ .appendFile(newDataFile("id=2/nested.id=2"))
+ .appendFile(newDataFile("id=3/nested.id=3"))
+ .commit();
+ }
+
+ @Test
+ public void testManifestsTableWithAddPartitionOnNestedField() throws IOException {
+ Table manifestsTable = new ManifestsTable(table.ops(), table);
+ TableScan scan = manifestsTable.newScan();
+
+ try (CloseableIterable<FileScanTask> tasks = scan.planFiles()) {
+ Assert.assertEquals("Should have one tasks", 1, Iterables.size(tasks));
Review Comment:
nit: I know this is copy/pasted/aligned with the other test, but I think it would be better to change those newly added assertions to `Assertions.assertThat(tasks).hasSize(1);` and the other one to `Assertions.assertThat(Streams.stream(tasks).mapToInt(task -> Iterables.size(task.asDataTask().rows()))).hasSize(2);`
Using AssertJ assertions will actually print the content of the iterable if the check ever fails, and thus will make debugging much easier. It would be great to adjust all of the assertions in this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1013697964
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -107,13 +107,15 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String fieldName = AvroSchemaUtil.makeCompatibleName(field.name());
Schema.Field newField =
new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())),
null,
JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ newField.addProp(AvroSchemaUtil.ICEBERG_FIELD_NAME_PROP, field.name());
Review Comment:
Changed to set the property `AvroSchemaUtil.ICEBERG_FIELD_NAME_PROP` only when the field name is sanitized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r852205263
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Review Comment:
Is there an issue with us just always calling sanitize?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851855921
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1009687200
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -107,13 +107,15 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String fieldName = AvroSchemaUtil.makeCompatibleName(field.name());
Schema.Field newField =
new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())),
null,
JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ newField.addProp(AvroSchemaUtil.ICEBERG_FIELD_NAME_PROP, field.name());
Review Comment:
Yea can we align with that code? like use origFieldName instead of field.name()?
``` field.addProp(AvroSchemaUtil.ICEBERG_FIELD_NAME_PROP, origFieldName); ```
I was also thinking we should align with that, in not setting it unless field name is changed? Otherwise it seems inconsistent behavior. @chenjunjiedada for reference in previous commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r852555002
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Schema.Field newField = new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())), null, JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ if (!isValidFieldName) {
Review Comment:
@RussellSpitzer @chenjunjiedada We need to determine whether the original field name prop needs to be set. This same as `BuildAvroProjection`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851855315
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
Review Comment:
This problem only happens on metadata table, because we use a common partition schema for metadata table. For example:
```scala
// create table
spark.sql("create table local.db.test_table(id long, data struct<id: long>) using iceberg partitioned by (identity(id))").show()
// manifest 0
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
// update table to add a partiition field on the nested field
spark.sql("alter table local.db.test_table add partition field identity(data.id)").show()
// manifest 1
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
spark.sql("select * from local.db.test_table.files").show()
```
The common partition schema is `(Identity(id), Identity(data.id))`. We used the common schema projecting on the manifest 0, we could not find the field(data.id) from the Avro schema which read from the manifest file. Then we need to create a new Avro field from the missing one which is why we reached here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851851785
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Review Comment:
extra space after '='
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r853767529
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Review Comment:
Updated to use `AvroSchemaUtil.makeCompatibleName`.
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Review Comment:
Updated to use `AvroSchemaUtil.makeCompatibleName`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1008532668
##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java:
##########
@@ -978,6 +1091,32 @@ private Set<String> expectedManifestListPaths(Iterable<Snapshot> snapshots, Long
.collect(Collectors.toSet());
}
+ private void createTableAndAddPartitionOnNestedFiled() throws IOException {
Review Comment:
Great to add all these tests. I think it may fit better in "TestMetadataTablesWithPartitionEvolution" because if i understand its a partition evolution case? But maybe its too much work to move the new tests there.
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -107,13 +107,15 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String fieldName = AvroSchemaUtil.makeCompatibleName(field.name());
Schema.Field newField =
new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())),
null,
JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ newField.addProp(AvroSchemaUtil.ICEBERG_FIELD_NAME_PROP, field.name());
Review Comment:
Nit: Can we use fieldName here? (Actually prefer to rename to 'oldFieldName' if you don't mind.
Also , do you know the significance of this property? Looks like in the other case its only set if the field name has been changed, should we do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1008683069
##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java:
##########
@@ -978,6 +1091,32 @@ private Set<String> expectedManifestListPaths(Iterable<Snapshot> snapshots, Long
.collect(Collectors.toSet());
}
+ private void createTableAndAddPartitionOnNestedFiled() throws IOException {
Review Comment:
OK, let me look it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#issuecomment-1308090077
Thanks @szehon-ho @nastra @chenjunjiedada @RussellSpitzer for your time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
nastra commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1011379701
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -107,13 +107,15 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String fieldName = AvroSchemaUtil.makeCompatibleName(field.name());
Review Comment:
this is already being done in L98, can we maybe do `String avroFieldName = AvroSchemaUtil.makeCompatibleName(field.name());` in L97 and use `avroFieldName` in L113?
##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java:
##########
@@ -978,6 +1091,32 @@ private Set<String> expectedManifestListPaths(Iterable<Snapshot> snapshots, Long
.collect(Collectors.toSet());
}
+ private void createTableAndAddPartitionOnNestedFiled() throws IOException {
Review Comment:
`TestMetadataTablesWithPartitionEvolution` seems to be a Spark test. I would actually prefer to have the tests in `iceberg-core` (maybe in a subclass of `TestReadProjection`?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho merged pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
szehon-ho merged PR #4577:
URL: https://github.com/apache/iceberg/pull/4577
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851855315
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
Review Comment:
This problem only happens on metadata table, because we use a common partition schema for metadata table. For example:
```scala
// create table
spark.sql("create table local.db.test_table(id long, data struct<id: long>) using iceberg partitioned by (identity(id))").show()
// manifest 0
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
// update table to add a partiition field on the nested field
spark.sql("alter table local.db.test_table add partition field identity(data.id)").show()
// manifest 1
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
spark.sql("select * from local.db.test_table.files").show()
```
The common partition schema is `(Identity(id), Identity(data.id))`. We use the common schema projecting on the manifest 0 when reading the manifest file, and we could not find the field(data.id) from the Avro schema which read from the manifest file. Then we need to create a new Avro field from the missing one which is why we reached here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851855315
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
Review Comment:
This problem only happens on metadata table, because we use a common partition schema for metadata table. For example:
```scala
// create table
spark.sql("create table local.db.test_table(id long, data struct<id: long>) using iceberg partitioned by (identity(id))").show()
// manifest 0
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
// update table to add a partiition field on the nested field
spark.sql("alter table local.db.test_table add partition field identity(data.id)").show()
// manifest 1
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
spark.sql("select * from local.db.test_table.files").show()
```
The common partition schema is `(Identity(id), Identity(data.id))`. We use the common schema projecting on the manifest 0 when reading the manifest file, and we could not find the field(data.id) from the Avro schema which read from the manifest file. Then we need to create a new Avro field from the missing one. And we met that problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r852637826
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Schema.Field newField = new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())), null, JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ if (!isValidFieldName) {
Review Comment:
We set according to comparing target `fieldName` and `originFiledName`, right? Also, can we always set the property to `originFieldName`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#issuecomment-1307316351
Merged, thanks @ConeyLiu , and also @nastra @chenjunjiedada @RussellSpitzer for reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r1013145015
##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScans.java:
##########
@@ -978,6 +1091,32 @@ private Set<String> expectedManifestListPaths(Iterable<Snapshot> snapshots, Long
.collect(Collectors.toSet());
}
+ private void createTableAndAddPartitionOnNestedFiled() throws IOException {
Review Comment:
Ah didnt realize, yea its fine then for me where it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851855315
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
Review Comment:
This problem only happens on metadata table, because we use a common partition schema for metadata table. For example:
```scala
// create table
spark.sql("create table local.db.test_table(id long, data struct<id: long>) using iceberg partitioned by (identity(id))").show()
// manifest 0
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
// update table to add a partiition field on the nested field
spark.sql("alter table local.db.test_table add partition field identity(data.id)").show()
// manifest 1
spark.range(10).map(i => Item(i, Data(i))).writeTo("local.db.test_table").append()
spark.sql("select * from local.db.test_table.files").show()
```
The common partition schema is `(Identity(id), Identity(data.id))` when we used the common schema projecting on the manifest 0, we could not find the field(data.id) from the Avro schema which read from the manifest file. Then we need to create a new Avro field from the missing one which is why we reached here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r852536270
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Review Comment:
Aha, +1 to just use `AvroSchemaUtil.sanitize(origFieldName)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ConeyLiu commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r852555002
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Schema.Field newField = new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())), null, JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ if (!isValidFieldName) {
Review Comment:
@RussellSpitzer @chenjunjiedada We need to determine whether the original field name prop needs to be set. This same as `TypeToSchema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] chenjunjiedada commented on a diff in pull request #4577: Fixes read metadata table failed due to illegal character
Posted by GitBox <gi...@apache.org>.
chenjunjiedada commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r852637826
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names, Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique suffix to the field
// to make sure that even if records in the file have the field it is not projected.
+ String origFieldName = field.name();
+ boolean isValidFieldName = AvroSchemaUtil.validAvroName(origFieldName);
+ String fieldName = isValidFieldName ? origFieldName : AvroSchemaUtil.sanitize(origFieldName);
Schema.Field newField = new Schema.Field(
- field.name() + "_r" + field.fieldId(),
+ fieldName + "_r" + field.fieldId(),
AvroSchemaUtil.toOption(AvroSchemaUtil.convert(field.type())), null, JsonProperties.NULL_VALUE);
newField.addProp(AvroSchemaUtil.FIELD_ID_PROP, field.fieldId());
+ if (!isValidFieldName) {
Review Comment:
We can set according to comparing target `fieldName` and `originFiledName`, right? Also, can we always set the property to `originFieldName`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org