You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/08/08 01:34:25 UTC
[GitHub] [iceberg] wypoon opened a new pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
wypoon opened a new pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955
The constructor of RepartitionByExpression changed between Spark 3.0 and 3.1.
There was an instance of constructing RepartitionByExpression that was missed in the original commit (#2512).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
Posted by GitBox <gi...@apache.org>.
wypoon commented on pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955#issuecomment-894729200
@RussellSpitzer we can use this if you'd like to have some tests that show the issue now, or we can use #2954 and wait for #2911 for such tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on a change in pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
Posted by GitBox <gi...@apache.org>.
wypoon commented on a change in pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955#discussion_r685620417
##########
File path: spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
##########
@@ -721,9 +723,25 @@ public void testDeleteRefreshesRelationCache() throws NoSuchTableException {
// TODO: multiple stripes for ORC
+ protected void setDistributionModeOnTable() {
+ DistributionMode distMode = DistributionMode.NONE;
+ switch (fileFormat) {
+ case "parquet":
+ distMode = DistributionMode.HASH;
+ break;
+ case "orc":
+ break;
+ case "avro":
+ distMode = DistributionMode.RANGE;
+ break;
+ }
+ sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')", tableName, WRITE_DISTRIBUTION_MODE, distMode);
Review comment:
@coolderli #2954 (which doesn't have test changes) got merged instead. I look forward to your completing #2911.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon closed pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
Posted by GitBox <gi...@apache.org>.
wypoon closed pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
Posted by GitBox <gi...@apache.org>.
wypoon commented on pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955#issuecomment-895567688
Abandoned in favor of #2954.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coolderli commented on a change in pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
Posted by GitBox <gi...@apache.org>.
coolderli commented on a change in pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955#discussion_r685224702
##########
File path: spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
##########
@@ -721,9 +723,25 @@ public void testDeleteRefreshesRelationCache() throws NoSuchTableException {
// TODO: multiple stripes for ORC
+ protected void setDistributionModeOnTable() {
+ DistributionMode distMode = DistributionMode.NONE;
+ switch (fileFormat) {
+ case "parquet":
+ distMode = DistributionMode.HASH;
+ break;
+ case "orc":
+ break;
+ case "avro":
+ distMode = DistributionMode.RANGE;
+ break;
+ }
+ sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')", tableName, WRITE_DISTRIBUTION_MODE, distMode);
Review comment:
@wypoon Maybe you can just move these modify in #2911 to this patch. It can also test `UPDATE` and `MERGE into` in different distribution modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coolderli commented on a change in pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
Posted by GitBox <gi...@apache.org>.
coolderli commented on a change in pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955#discussion_r685224935
##########
File path: spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
##########
@@ -721,9 +723,25 @@ public void testDeleteRefreshesRelationCache() throws NoSuchTableException {
// TODO: multiple stripes for ORC
+ protected void setDistributionModeOnTable() {
+ DistributionMode distMode = DistributionMode.NONE;
+ switch (fileFormat) {
+ case "parquet":
+ distMode = DistributionMode.HASH;
+ break;
+ case "orc":
+ break;
+ case "avro":
+ distMode = DistributionMode.RANGE;
+ break;
+ }
+ sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')", tableName, WRITE_DISTRIBUTION_MODE, distMode);
Review comment:
@wypoon Maybe you can just move these modify in #2911 to this patch. It can also test `UPDATE` and `MERGE into` in different distribution modes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on a change in pull request #2955: [Spark] Support building against both Spark 3.0 and Spark 3.1: follow-up
Posted by GitBox <gi...@apache.org>.
wypoon commented on a change in pull request #2955:
URL: https://github.com/apache/iceberg/pull/2955#discussion_r684699703
##########
File path: spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestDelete.java
##########
@@ -721,9 +723,25 @@ public void testDeleteRefreshesRelationCache() throws NoSuchTableException {
// TODO: multiple stripes for ORC
+ protected void setDistributionModeOnTable() {
+ DistributionMode distMode = DistributionMode.NONE;
+ switch (fileFormat) {
+ case "parquet":
+ distMode = DistributionMode.HASH;
+ break;
+ case "orc":
+ break;
+ case "avro":
+ distMode = DistributionMode.RANGE;
+ break;
+ }
+ sql("ALTER TABLE %s SET TBLPROPERTIES('%s' '%s')", tableName, WRITE_DISTRIBUTION_MODE, distMode);
Review comment:
This produces the same matrix of parameters as in #2911, but only for the partitioned tables used in TestDelete and its subclass(es).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org