You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2023/04/07 23:38:43 UTC
[GitHub] [iceberg] amogh-jahagirdar opened a new pull request, #7300: Spark: Show Create Round trip tests
amogh-jahagirdar opened a new pull request, #7300:
URL: https://github.com/apache/iceberg/pull/7300
Follow up for https://github.com/apache/iceberg/pull/7273 . we are missing `SHOW CREATE` tests for Spark. I'm adding these in TestCreateTable so that we can test round trip behavior for a variety of cases (partitioning, column comments, table comments, table properties etc).
Let me know what you think @RussellSpitzer @aokolnychyi @jackye1995 @rdblue !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161038572
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
>For me, show create is working correctly if I can identically recreate the original table. The actual string is kind of inconsequential and given the fragility of string syntaxes over time, we probably don't want to rely on that.
Right I think this is the key point for why string checking is probably not the right way, I'm looking into table equality, but overall I got the idea. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161037072
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Let me elaborate a bit. First we make the table, load it in iceberg and keep this object in memory. We could use "SerializableTable" or something like that. Then we should make the table again with the show create output. The new table should have equality on all components with the original. Same name, schema, partition spec, comments, properties, etc ... I'm not sure we fully spec'd out object.equals for Tables, but if we did this would be a good use for that. I'm on break at the moment but can check this out more when I get back.
For me, show create is working correctly if I can identically recreate the original table. The actual string is kind of inconsequential and given the fragility of string syntaxes over time, we probably don't want to rely on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161026208
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
This parts still a bit messy. I'm open to suggestions here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161038572
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
>For me, show create is working correctly if I can identically recreate the original table. The actual string is kind of inconsequential and given the fragility of string syntaxes over time, we probably don't want to rely on that.
Right I think this is the key point, I'm looking into table equality
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161026208
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
This parts still messy. I'm open to suggestions here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161035147
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Yeah I considered this, I guess I was thinking for these tests we want to validate what is the Spark-iCeberg user experience (the actual string they get back when they do show create and make sure that is as expected). if we test the actual table entity we don't get that but we do still validate the core table structure is as expected, which should be sufficient. I'll update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161035147
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Yeah I considered this, I guess I was thinking for these tests we want to validate what is the Spark-iCeberg user experience (the actual string they get back when they do show create and make sure that is as expected). if we test the actual table entity we don't get that but we do still validate the core table structure is as expected, which should be sufficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161866110
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Cool will update. IT looks like table equality isn't formally defined yet, but at least for purpose of this test we can add our own validation for the relevant table fields.
Also one thing to note is that Spark Show create tests https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala perform a string check, but I don't think we need to follow that here because for Iceberg we have more combinations that will make the string fragile
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Cool will update. It looks like table equality isn't formally defined yet, but at least for purpose of this test we can add our own validation for the relevant table fields.
Also one thing to note is that Spark Show create tests https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala perform a string check, but I don't think we need to follow that here because for Iceberg we have more combinations that will make the string fragile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161031536
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
I would try not to do any string checking at all. We should be able to just drop the table, call create on the show create statement and get back an object with all the same schema and properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161035147
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Yeah I considered this, I guess I was thinking for these tests we want to validate what is the Spark-iCeberg user experience (the actual create table a user gets when they do show create and make sure that string is as expected and remains consistent). if we test the actual table entity we don't get that but we do still validate the core table structure is consistent, which should be sufficient. I'll update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161037072
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Let me elaborate a bit. First we make the table, load it in iceberg and keep this object in memory. We could use "SerializableTable" or something like that. Then we should make the table again with the show create output. The new table should have equality on all components with the original. Same name, schema, partition spec, comments, properties, etc ... I'm not sure we fully spec'd out object.equals for Tables, but if we did this would be a good use for that. I'm on break at the moment but can check this out more when I get back.
For me, show create is working correctly if I can identically recreated the table I had before. The actual string is kind of inconsequential and given the fragility of string syntaxes over time, we probably don't want to rely on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161031755
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
For example
Create table A ...
X = show create table a
Drop a
Sql(x)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161869535
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
> It looks like table equality isn't formally defined yet
yes I think that's the current status. Maybe we can take this chance to add definition of table equality?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "jackye1995 (via GitHub)" <gi...@apache.org>.
jackye1995 commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161851380
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
+1, string check is too fragile.
It might be tricky to do full comparison of the table though, we have methods like `Schema.sameSchema` to check schema equality, maybe we should fill in the missing equality check methods for all relevant components of the table, so we can do a `table.sameTable(anotherTable)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161026208
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
This parts still messy. I'm open to suggestions here! A lot of the messiness is derived from trying to do an exact String match with show create, maybe if we just normalize the string we can get away from having exact expectations of newlines and spacing etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on a diff in pull request #7300: Spark: Show Create Round trip tests
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #7300:
URL: https://github.com/apache/iceberg/pull/7300#discussion_r1161035147
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestCreateTable.java:
##########
@@ -341,4 +414,65 @@ public void testDowngradeTableToFormatV1ThroughTablePropertyFails() {
"Cannot downgrade v2 table to v1",
() -> sql("ALTER TABLE %s SET TBLPROPERTIES ('format-version'='1')", tableName));
}
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, null);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String comment) {
+ validateExpectedShowCreateTable(
+ tableName, expectedSchema, expectedLocation, expectedProperties, null, null, comment);
+ }
+
+ private void validateExpectedShowCreateTable(
+ String tableName,
+ String expectedSchema,
+ String expectedLocation,
+ Map<String, String> expectedProperties,
+ String partitionClause,
+ String bucketClause,
+ String comment) {
+ StringBuilder expectedCreate = new StringBuilder();
+
+ expectedCreate.append(
+ String.format(
+ "CREATE TABLE %s ",
+ (catalogName.equals("spark_catalog") ? "spark_catalog." : "") + tableName));
+
+ expectedCreate.append(expectedSchema + "\n");
+ expectedCreate.append("USING iceberg\n");
+ if (comment != null) {
+ expectedCreate.append(comment + "\n");
+ }
+
+ if (partitionClause != null) {
+ expectedCreate.append(partitionClause + "\n");
+ }
+
+ if (bucketClause != null) {
+ expectedCreate.append(bucketClause + "\n");
+ }
+
+ expectedCreate.append(String.format("LOCATION '%s'\n", expectedLocation));
+ expectedCreate.append("TBLPROPERTIES (\n ");
+ expectedCreate.append(tablePropsAsString(expectedProperties, " = ", ",\n "));
+ expectedCreate.append(")\n");
+
+ List<Object[]> actualCreate = sql("SHOW CREATE table %s", tableName);
+
+ assertEquals(
+ "Should have expected create",
+ ImmutableList.of(row(expectedCreate.toString())),
+ actualCreate);
Review Comment:
Yeah I considered this, I guess I was thinking for these tests we want to validate what is the Spark-iCeberg user experience (the actual create table a user gets when they do show create and make sure that is as expected and remains consistent). if we test the actual table entity we don't get that but we do still validate the core table structure is as expected, which should be sufficient. I'll update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org