You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/08/29 00:24:59 UTC
[GitHub] [iceberg] edgarRd opened a new pull request #1399: Spark: Follow name mapping when importing ORC tables
edgarRd opened a new pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399
This PR fixes #1345 and #1206 - the main changes included are:
1. Refactor `TestSparkTableUtil` to run tests in multiple file formats.
2. Use name mapping when importing ORC tables.
Note that I changed to use a name mapping by default of the target table, as mentioned in https://github.com/apache/iceberg/issues/1345#issuecomment-679417869.
PTAL @rdsr @shardulm94 @aokolnychyi - Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-689127017
Thanks for pinging me, @edgarRd. I'm back from a long weekend off and I'll look as soon as I have time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-699135771
@edgarRd, I had another look. The default is still changed so that a table with no mapping with create one. That needs to be reverted before we can commit this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r487601001
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestSparkTableUtil.java
##########
@@ -47,32 +47,31 @@
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Assert;
+import org.junit.Assume;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
import static org.apache.iceberg.TableProperties.PARQUET_VECTORIZATION_ENABLED;
import static org.apache.iceberg.types.Types.NestedField.optional;
+@RunWith(Enclosed.class)
Review comment:
While there are a lot of changes in this test, can you also move this to iceberg-spark instead of iceberg-spark2? I think it is in spark2 by accident, and moving it in IntelliJ produces no warnings. That way this also runs in the Java 11 test profile.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r495208620
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestSparkTableUtil.java
##########
@@ -47,32 +47,31 @@
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Assert;
+import org.junit.Assume;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
import static org.apache.iceberg.TableProperties.PARQUET_VECTORIZATION_ENABLED;
import static org.apache.iceberg.types.Types.NestedField.optional;
+@RunWith(Enclosed.class)
Review comment:
Yeah, that's fine with me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r495211519
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java
##########
@@ -573,7 +574,8 @@ public static void importSparkPartitions(
int numShufflePartitions = spark.sessionState().conf().numShufflePartitions();
MetricsConfig metricsConfig = MetricsConfig.fromProperties(targetTable.properties());
String nameMappingString = targetTable.properties().get(TableProperties.DEFAULT_NAME_MAPPING);
- NameMapping nameMapping = nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ NameMapping nameMapping = nameMappingString != null ?
+ NameMappingParser.fromJson(nameMappingString) : MappingUtil.create(targetTable.schema());
Review comment:
This change needs to be reverted. The import process must create and store a mapping on the table. We should not use an ephemeral mapping that is immediately discarded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r495251583
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java
##########
@@ -573,7 +574,8 @@ public static void importSparkPartitions(
int numShufflePartitions = spark.sessionState().conf().numShufflePartitions();
MetricsConfig metricsConfig = MetricsConfig.fromProperties(targetTable.properties());
String nameMappingString = targetTable.properties().get(TableProperties.DEFAULT_NAME_MAPPING);
- NameMapping nameMapping = nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ NameMapping nameMapping = nameMappingString != null ?
+ NameMappingParser.fromJson(nameMappingString) : MappingUtil.create(targetTable.schema());
Review comment:
Yeah, somehow I missed this one but did it for the method of unpartitioned tables. Thanks for pointing this out. I've reverted the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
edgarRd commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-698580233
@rdblue I've updated the comment on the default. I was wondering if the relocation of the tests could be done in a follow up PR just to make the diff clear. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-691752808
Great work, @edgarRd! Everything looks good, except for changing the default behavior. Let's separate that out and discuss how we want to do it. Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-699178200
Thanks for updating! I'll merge this when tests are passing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r494600713
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestSparkTableUtil.java
##########
@@ -47,32 +47,31 @@
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Assert;
+import org.junit.Assume;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
import static org.apache.iceberg.TableProperties.PARQUET_VECTORIZATION_ENABLED;
import static org.apache.iceberg.types.Types.NestedField.optional;
+@RunWith(Enclosed.class)
Review comment:
I agree. I could work on another PR with that change, I was wondering if that'd minimize the diff in this PR and make it more clear. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
edgarRd commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-698580233
@rdblue I've updated the comment on the default. I was wondering if the relocation of the tests could be done in a follow up PR just to make the diff clear. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
edgarRd commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r494600713
##########
File path: spark2/src/test/java/org/apache/iceberg/spark/source/TestSparkTableUtil.java
##########
@@ -47,32 +47,31 @@
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Assert;
+import org.junit.Assume;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
import org.junit.rules.TemporaryFolder;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
import static org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING;
import static org.apache.iceberg.TableProperties.PARQUET_VECTORIZATION_ENABLED;
import static org.apache.iceberg.types.Types.NestedField.optional;
+@RunWith(Enclosed.class)
Review comment:
I agree. I could work on another PR with that change, I was wondering if that'd minimize the diff in this PR and make it more clear. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r495269405
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java
##########
@@ -573,7 +574,8 @@ public static void importSparkPartitions(
int numShufflePartitions = spark.sessionState().conf().numShufflePartitions();
MetricsConfig metricsConfig = MetricsConfig.fromProperties(targetTable.properties());
String nameMappingString = targetTable.properties().get(TableProperties.DEFAULT_NAME_MAPPING);
- NameMapping nameMapping = nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ NameMapping nameMapping = nameMappingString != null ?
+ NameMappingParser.fromJson(nameMappingString) : MappingUtil.create(targetTable.schema());
Review comment:
That explains it. I thought that you were going to, so I was surprised.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
edgarRd commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-699168313
@rdblue Thanks for taking a look!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] edgarRd commented on pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
edgarRd commented on pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#issuecomment-686155295
@rdsr @shardulm94 @aokolnychyi @rdblue PTAL if you have a chance. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1399: Spark: Follow name mapping when importing ORC tables
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1399:
URL: https://github.com/apache/iceberg/pull/1399#discussion_r487596377
##########
File path: spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java
##########
@@ -540,7 +541,8 @@ private static void importUnpartitionedSparkTable(
Configuration conf = spark.sessionState().newHadoopConf();
MetricsConfig metricsConfig = MetricsConfig.fromProperties(targetTable.properties());
String nameMappingString = targetTable.properties().get(TableProperties.DEFAULT_NAME_MAPPING);
- NameMapping nameMapping = nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;
+ NameMapping nameMapping = nameMappingString != null ?
+ NameMappingParser.fromJson(nameMappingString) : MappingUtil.create(targetTable.schema());
Review comment:
I think we should change to use a name mapping by default in a separate PR. For now, let's just support a name mapping when importing ORC data.
My rationale is that I think the two are independently useful. Someone might want to change the default, but not cherry-pick ORC changes. Similarly, someone might want to use a name mapping with ORC, but not want to default to name mapping.
Also, I think that we would want to default to name mapping slightly differently. It doesn't make sense to me to create a temporary mapping that is used for metrics here, unless that mapping is also used to read the data. So I would prefer to update tables when importing and add a mapping to table metadata by default, if it is not already there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org