You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/01/18 16:22:45 UTC
[GitHub] [iceberg] aokolnychyi opened a new pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
aokolnychyi opened a new pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918
This PR adds tests that validate the resolution of star actions is done by name, which was a recent breaking change in Spark.
The credit goes to @kbendick, the original author of the patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918#issuecomment-1015796996
Oh I have an open PR for this haha. We can use this one though since it's reviewed. https://github.com/apache/iceberg/pull/3440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918#discussion_r787067750
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -1111,6 +1111,83 @@ public void testMergeRefreshesRelationCache() {
spark.sql("UNCACHE TABLE tmp");
}
+ @Test
+ public void testMergeResolvesColumnsByName() {
+ createAndInitTable("id INT, badge INT, dep STRING",
+ "{ \"id\": 1, \"badge\": 1000, \"dep\": \"emp-id-one\" }\n" +
+ "{ \"id\": 6, \"badge\": 6000, \"dep\": \"emp-id-6\" }");
+
+ createOrReplaceView("source", "badge INT, id INT, dep STRING",
+ "{ \"badge\": 1001, \"id\": 1, \"dep\": \"emp-id-1\" }\n" +
+ "{ \"badge\": 6006, \"id\": 6, \"dep\": \"emp-id-6\" }\n" +
+ "{ \"badge\": 7007, \"id\": 7, \"dep\": \"emp-id-7\" }");
+
+ sql("MERGE INTO %s AS t USING source AS s " +
+ "ON t.id == s.id " +
+ "WHEN MATCHED THEN " +
+ " UPDATE SET * " +
+ "WHEN NOT MATCHED THEN " +
+ " INSERT * ", tableName);
+
+ ImmutableList<Object[]> expectedRows = ImmutableList.of(
+ row(1, 1001, "emp-id-1"), // updated
+ row(6, 6006, "emp-id-6"), // updated
+ row(7, 7007, "emp-id-7") // new
+ );
+ assertEquals("Should have expected rows", expectedRows,
+ sql("SELECT id, badge, dep FROM %s ORDER BY id", tableName));
+ }
+
+ @Test
+ public void testMergeWithLiteralBooleanAsPredicate() {
+ createAndInitTable("id INT, dep STRING");
+
+ createOrReplaceView("source", "id INT, dep STRING",
+ "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n" +
+ "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n" +
+ "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
+
+ sql("MERGE INTO %s AS t USING source AS s " +
+ "ON true " +
Review comment:
After a closer look, it does seem weird. It works because there are no table rows but I think we can remove this test. I also adapted the test below and inverted the condition so that it's more reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918#issuecomment-1015801487
Missed the other PR. Merged this one as @kbendick is a co-author and it has been reviewed.
Thanks, @rdblue @kbendick!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918#discussion_r786985572
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -1111,6 +1111,83 @@ public void testMergeRefreshesRelationCache() {
spark.sql("UNCACHE TABLE tmp");
}
+ @Test
+ public void testMergeResolvesColumnsByName() {
+ createAndInitTable("id INT, badge INT, dep STRING",
+ "{ \"id\": 1, \"badge\": 1000, \"dep\": \"emp-id-one\" }\n" +
+ "{ \"id\": 6, \"badge\": 6000, \"dep\": \"emp-id-6\" }");
+
+ createOrReplaceView("source", "badge INT, id INT, dep STRING",
+ "{ \"badge\": 1001, \"id\": 1, \"dep\": \"emp-id-1\" }\n" +
+ "{ \"badge\": 6006, \"id\": 6, \"dep\": \"emp-id-6\" }\n" +
+ "{ \"badge\": 7007, \"id\": 7, \"dep\": \"emp-id-7\" }");
+
+ sql("MERGE INTO %s AS t USING source AS s " +
+ "ON t.id == s.id " +
+ "WHEN MATCHED THEN " +
+ " UPDATE SET * " +
+ "WHEN NOT MATCHED THEN " +
+ " INSERT * ", tableName);
+
+ ImmutableList<Object[]> expectedRows = ImmutableList.of(
+ row(1, 1001, "emp-id-1"), // updated
+ row(6, 6006, "emp-id-6"), // updated
+ row(7, 7007, "emp-id-7") // new
+ );
+ assertEquals("Should have expected rows", expectedRows,
+ sql("SELECT id, badge, dep FROM %s ORDER BY id", tableName));
+ }
+
+ @Test
+ public void testMergeWithLiteralBooleanAsPredicate() {
+ createAndInitTable("id INT, dep STRING");
+
+ createOrReplaceView("source", "id INT, dep STRING",
+ "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n" +
+ "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n" +
+ "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
+
+ sql("MERGE INTO %s AS t USING source AS s " +
+ "ON true " +
Review comment:
Why does this work? Because there are no table rows?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi merged pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
Posted by GitBox <gi...@apache.org>.
aokolnychyi merged pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918#discussion_r787067750
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -1111,6 +1111,83 @@ public void testMergeRefreshesRelationCache() {
spark.sql("UNCACHE TABLE tmp");
}
+ @Test
+ public void testMergeResolvesColumnsByName() {
+ createAndInitTable("id INT, badge INT, dep STRING",
+ "{ \"id\": 1, \"badge\": 1000, \"dep\": \"emp-id-one\" }\n" +
+ "{ \"id\": 6, \"badge\": 6000, \"dep\": \"emp-id-6\" }");
+
+ createOrReplaceView("source", "badge INT, id INT, dep STRING",
+ "{ \"badge\": 1001, \"id\": 1, \"dep\": \"emp-id-1\" }\n" +
+ "{ \"badge\": 6006, \"id\": 6, \"dep\": \"emp-id-6\" }\n" +
+ "{ \"badge\": 7007, \"id\": 7, \"dep\": \"emp-id-7\" }");
+
+ sql("MERGE INTO %s AS t USING source AS s " +
+ "ON t.id == s.id " +
+ "WHEN MATCHED THEN " +
+ " UPDATE SET * " +
+ "WHEN NOT MATCHED THEN " +
+ " INSERT * ", tableName);
+
+ ImmutableList<Object[]> expectedRows = ImmutableList.of(
+ row(1, 1001, "emp-id-1"), // updated
+ row(6, 6006, "emp-id-6"), // updated
+ row(7, 7007, "emp-id-7") // new
+ );
+ assertEquals("Should have expected rows", expectedRows,
+ sql("SELECT id, badge, dep FROM %s ORDER BY id", tableName));
+ }
+
+ @Test
+ public void testMergeWithLiteralBooleanAsPredicate() {
+ createAndInitTable("id INT, dep STRING");
+
+ createOrReplaceView("source", "id INT, dep STRING",
+ "{ \"id\": 1, \"dep\": \"emp-id-1\" }\n" +
+ "{ \"id\": 2, \"dep\": \"emp-id-2\" }\n" +
+ "{ \"id\": 3, \"dep\": \"emp-id-3\" }");
+
+ sql("MERGE INTO %s AS t USING source AS s " +
+ "ON true " +
Review comment:
After a closer look, it does look weird. It works because there are no table rows but I think we can remove this test. I also adapted the test below and inverted the condition so that it's more reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #3918: Spark 3.2: Add tests for resolving star actions in MERGE by name
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #3918:
URL: https://github.com/apache/iceberg/pull/3918#issuecomment-1015765159
Looks good now. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org