You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/11 13:30:31 UTC
[GitHub] [spark] MaxGekk opened a new pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
MaxGekk opened a new pull request #31131:
URL: https://github.com/apache/spark/pull/31131
### What changes were proposed in this pull request?
Port the test added by https://github.com/apache/spark/pull/31112 to:
1. v1 In-Memory catalog for `ALTER TABLE .. DROP PARTITION`
2. v1 In-Memory and Hive external catalogs for `ALTER TABLE .. ADD PARTITION`
3. v1 In-Memory and Hive external catalogs for `ALTER TABLE .. RENAME PARTITION`
### Why are the changes needed?
To improve test coverage.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By running the modified test suites:
```
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableAddPartitionSuite"
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableDropPartitionSuite"
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.AlterTableRenamePartitionSuite"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-757956451
**[Test build #133930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133930/testReport)** for PR 31131 at commit [`1eb093e`](https://github.com/apache/spark/commit/1eb093e76b4331a49b23cba39bd155a25b547fa6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31131:
URL: https://github.com/apache/spark/pull/31131#discussion_r555280134
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableDropPartitionSuite.scala
##########
@@ -43,6 +44,28 @@ trait AlterTableDropPartitionSuiteBase extends command.AlterTableDropPartitionSu
checkPartitions(t) // no partitions
}
}
+
+ test("SPARK-34060, SPARK-34071: update stats of cached table") {
+ withSQLConf(SQLConf.AUTO_SIZE_UPDATE_ENABLED.key -> "true") {
+ withNamespaceAndTable("ns", "tbl") { t =>
+ sql(s"CREATE TABLE $t (id int, part int) $defaultUsing PARTITIONED BY (part)")
+ sql(s"INSERT INTO $t PARTITION (part=0) SELECT 0")
+ sql(s"INSERT INTO $t PARTITION (part=1) SELECT 1")
+ assert(!spark.catalog.isCached(t))
+ sql(s"CACHE TABLE $t")
+ assert(spark.catalog.isCached(t))
+ checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 0), Row(1, 1)))
+ val twoPartSize = getTableSize(t)
+ assert(twoPartSize > 0)
+
+ sql(s"ALTER TABLE $t DROP PARTITION (part=0)")
+ assert(spark.catalog.isCached(t))
+ val onePartSize = getTableSize(t)
+ assert(0 < onePartSize && onePartSize < twoPartSize)
Review comment:
> does getTableSize look at cached table for stats
No, it doesn't but updating stats uncached table before the PR https://github.com/apache/spark/pull/31112
Actually, I moved the test from Hive specific test suite to the base test suite for v1 catalogs. Here, I made the test portable because table data has different sizes, so, instead of comparing exact numbers, I replaced that by this check (which is independent from table size).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-759080784
**[Test build #133977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133977/testReport)** for PR 31131 at commit [`4153ba2`](https://github.com/apache/spark/commit/4153ba2b254756fb7d865e8746d48cc2defd1e91).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758562588
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38549/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31131:
URL: https://github.com/apache/spark/pull/31131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758538159
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38549/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31131:
URL: https://github.com/apache/spark/pull/31131#discussion_r555569367
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala
##########
@@ -91,4 +91,20 @@ trait DDLCommandTestUtils extends SQLTestUtils {
}
protected def checkLocation(t: String, spec: TablePartitionSpec, expected: String): Unit
+
+ // Getting the total table size in the filesystem in bytes
+ def getTableSize(tableName: String): Int = {
+ val stats =
+ sql(s"DESCRIBE TABLE EXTENDED $tableName")
+ .select("data_type")
+ .where("col_name = 'Statistics'")
Review comment:
nit: let's call `where` before `select`, otherwise the query looks weird.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758824886
**[Test build #133977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133977/testReport)** for PR 31131 at commit [`4153ba2`](https://github.com/apache/spark/commit/4153ba2b254756fb7d865e8746d48cc2defd1e91).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758520167
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38549/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758562588
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38549/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758005833
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38517/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758265246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31131:
URL: https://github.com/apache/spark/pull/31131#discussion_r555569367
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala
##########
@@ -91,4 +91,20 @@ trait DDLCommandTestUtils extends SQLTestUtils {
}
protected def checkLocation(t: String, spec: TablePartitionSpec, expected: String): Unit
+
+ // Getting the total table size in the filesystem in bytes
+ def getTableSize(tableName: String): Int = {
+ val stats =
+ sql(s"DESCRIBE TABLE EXTENDED $tableName")
+ .select("data_type")
+ .where("col_name = 'Statistics'")
Review comment:
nit: let's call `where` before `select`, otherwise the query looks weird. There should be no `col_name` column after `select("data_type")`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31131:
URL: https://github.com/apache/spark/pull/31131#discussion_r555241674
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableAddPartitionSuite.scala
##########
@@ -46,6 +47,24 @@ trait AlterTableAddPartitionSuiteBase extends command.AlterTableAddPartitionSuit
}
}
+ private def copyPartition(t: String, from: String, to: String): String = {
Review comment:
ditto: rename `t` (and perhaps `from` and `to` as well) to something more meaningful
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala
##########
@@ -91,4 +91,20 @@ trait DDLCommandTestUtils extends SQLTestUtils {
}
protected def checkLocation(t: String, spec: TablePartitionSpec, expected: String): Unit
+
+ // Getting the total table size in the filesystem in bytes
+ def getTableSize(t: String): Int = {
Review comment:
nit: Should we use `Long` as return type? yeah I know it's supposed to be only in test but just to be on the safe side.
Also perhaps rename `t` to `tableName` or something so it is easier to read. :)
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableRenamePartitionSuite.scala
##########
@@ -46,6 +47,27 @@ trait AlterTableRenamePartitionSuiteBase extends command.AlterTableRenamePartiti
checkAnswer(sql(s"SELECT id, data FROM $t WHERE id = 3"), Row(3, "def"))
}
}
+
+ test("SPARK-34060, SPARK-34071: update stats of cached table") {
+ withSQLConf(SQLConf.AUTO_SIZE_UPDATE_ENABLED.key -> "true") {
+ withNamespaceAndTable("ns", "tbl") { t =>
+ sql(s"CREATE TABLE $t (id int, part int) $defaultUsing PARTITIONED BY (part)")
+ sql(s"INSERT INTO $t PARTITION (part=0) SELECT 0")
+ sql(s"INSERT INTO $t PARTITION (part=1) SELECT 1")
+ assert(!spark.catalog.isCached(t))
+ sql(s"CACHE TABLE $t")
+ assert(spark.catalog.isCached(t))
+ QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 0), Row(1, 1)))
+ val tableSize = getTableSize(t)
+ assert(tableSize > 0)
+
+ sql(s"ALTER TABLE $t PARTITION (part=0) RENAME TO PARTITION (part=2)")
+ assert(spark.catalog.isCached(t))
+ assert(tableSize == getTableSize(t))
+ QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 2), Row(1, 1)))
Review comment:
nit: not sure why need to have `QueryTest` here - I thought the trait already extends it?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableDropPartitionSuite.scala
##########
@@ -43,6 +44,28 @@ trait AlterTableDropPartitionSuiteBase extends command.AlterTableDropPartitionSu
checkPartitions(t) // no partitions
}
}
+
+ test("SPARK-34060, SPARK-34071: update stats of cached table") {
+ withSQLConf(SQLConf.AUTO_SIZE_UPDATE_ENABLED.key -> "true") {
+ withNamespaceAndTable("ns", "tbl") { t =>
+ sql(s"CREATE TABLE $t (id int, part int) $defaultUsing PARTITIONED BY (part)")
+ sql(s"INSERT INTO $t PARTITION (part=0) SELECT 0")
+ sql(s"INSERT INTO $t PARTITION (part=1) SELECT 1")
+ assert(!spark.catalog.isCached(t))
+ sql(s"CACHE TABLE $t")
+ assert(spark.catalog.isCached(t))
+ checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 0), Row(1, 1)))
+ val twoPartSize = getTableSize(t)
+ assert(twoPartSize > 0)
+
+ sql(s"ALTER TABLE $t DROP PARTITION (part=0)")
+ assert(spark.catalog.isCached(t))
+ val onePartSize = getTableSize(t)
+ assert(0 < onePartSize && onePartSize < twoPartSize)
Review comment:
hmm I'm not sure if this check is useful - seems it is unrelated to caching (does `getTableSize` look at cached table for stats?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758885690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38565/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-757956451
**[Test build #133930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133930/testReport)** for PR 31131 at commit [`1eb093e`](https://github.com/apache/spark/commit/1eb093e76b4331a49b23cba39bd155a25b547fa6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758311653
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133937/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758638464
**[Test build #133962 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133962/testReport)** for PR 31131 at commit [`1e5b96a`](https://github.com/apache/spark/commit/1e5b96a72e354bd730d031195949ab2595b9e55a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758496755
**[Test build #133962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133962/testReport)** for PR 31131 at commit [`1e5b96a`](https://github.com/apache/spark/commit/1e5b96a72e354bd730d031195949ab2595b9e55a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758176791
**[Test build #133937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133937/testReport)** for PR 31131 at commit [`543454b`](https://github.com/apache/spark/commit/543454ba357a4c9ee66823f333b7968951b207cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758128683
**[Test build #133930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133930/testReport)** for PR 31131 at commit [`1eb093e`](https://github.com/apache/spark/commit/1eb093e76b4331a49b23cba39bd155a25b547fa6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758206159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38523/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-759205547
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758299974
**[Test build #133937 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133937/testReport)** for PR 31131 at commit [`543454b`](https://github.com/apache/spark/commit/543454ba357a4c9ee66823f333b7968951b207cc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758265246
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-757983514
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38517/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758496755
**[Test build #133962 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133962/testReport)** for PR 31131 at commit [`1e5b96a`](https://github.com/apache/spark/commit/1e5b96a72e354bd730d031195949ab2595b9e55a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758248561
**[Test build #133935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133935/testReport)** for PR 31131 at commit [`e5aabf3`](https://github.com/apache/spark/commit/e5aabf3f36798056356374d961e85c452de72fcc).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758138172
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133930/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758233559
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758099892
@dongjoon-hyun @sunchao @HyukjinKwon Could you review this PR, please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758851694
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38565/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758655995
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133962/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31131:
URL: https://github.com/apache/spark/pull/31131#discussion_r555266380
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLCommandTestUtils.scala
##########
@@ -91,4 +91,20 @@ trait DDLCommandTestUtils extends SQLTestUtils {
}
protected def checkLocation(t: String, spec: TablePartitionSpec, expected: String): Unit
+
+ // Getting the total table size in the filesystem in bytes
+ def getTableSize(t: String): Int = {
Review comment:
We discussed this with @cloud-fan in other PRs, and decided to use short names for variables with small scope.
If you look at the unified tests, you can see `t` everywhere used for table names. This made sql statements shorter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758201175
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38523/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758023155
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38517/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758176791
**[Test build #133937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133937/testReport)** for PR 31131 at commit [`543454b`](https://github.com/apache/spark/commit/543454ba357a4c9ee66823f333b7968951b207cc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758655995
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133962/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758175843
**[Test build #133935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133935/testReport)** for PR 31131 at commit [`e5aabf3`](https://github.com/apache/spark/commit/e5aabf3f36798056356374d961e85c452de72fcc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758138172
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133930/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-759081706
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133977/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758885690
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38565/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758206159
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38523/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-759081706
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133977/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758158766
oh, sorry, I mixed two PRs here. Pushed unrelated changes. Let me revert last commits.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758824886
**[Test build #133977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133977/testReport)** for PR 31131 at commit [`4153ba2`](https://github.com/apache/spark/commit/4153ba2b254756fb7d865e8746d48cc2defd1e91).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758206129
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38523/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758448256
@HyukjinKwon @cloud-fan Could you look at this PR, please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758869946
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38565/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758175843
**[Test build #133935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133935/testReport)** for PR 31131 at commit [`e5aabf3`](https://github.com/apache/spark/commit/e5aabf3f36798056356374d961e85c452de72fcc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758311653
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133937/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31131:
URL: https://github.com/apache/spark/pull/31131#discussion_r555271036
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableRenamePartitionSuite.scala
##########
@@ -46,6 +47,27 @@ trait AlterTableRenamePartitionSuiteBase extends command.AlterTableRenamePartiti
checkAnswer(sql(s"SELECT id, data FROM $t WHERE id = 3"), Row(3, "def"))
}
}
+
+ test("SPARK-34060, SPARK-34071: update stats of cached table") {
+ withSQLConf(SQLConf.AUTO_SIZE_UPDATE_ENABLED.key -> "true") {
+ withNamespaceAndTable("ns", "tbl") { t =>
+ sql(s"CREATE TABLE $t (id int, part int) $defaultUsing PARTITIONED BY (part)")
+ sql(s"INSERT INTO $t PARTITION (part=0) SELECT 0")
+ sql(s"INSERT INTO $t PARTITION (part=1) SELECT 1")
+ assert(!spark.catalog.isCached(t))
+ sql(s"CACHE TABLE $t")
+ assert(spark.catalog.isCached(t))
+ QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 0), Row(1, 1)))
+ val tableSize = getTableSize(t)
+ assert(tableSize > 0)
+
+ sql(s"ALTER TABLE $t PARTITION (part=0) RENAME TO PARTITION (part=2)")
+ assert(spark.catalog.isCached(t))
+ assert(tableSize == getTableSize(t))
+ QueryTest.checkAnswer(sql(s"SELECT * FROM $t"), Seq(Row(0, 2), Row(1, 1)))
Review comment:
This is a workaround for the issue which I don't know how to resolve something else. If remove `QueryTest`, I am getting the error:
```
An exception or error caused a run to abort: Method org/apache/spark/sql/hive/execution/command/AlterTableRenamePartitionSuite.org$apache$spark$sql$execution$command$v1$AlterTableRenamePartitionSuiteBase$$super$checkAnswer(Lscala/Function0;Lscala/collection/Seq;)V is abstract
java.lang.AbstractMethodError: Method org/apache/spark/sql/hive/execution/command/AlterTableRenamePartitionSuite.org$apache$spark$sql$execution$command$v1$AlterTableRenamePartitionSuiteBase$$super$checkAnswer(Lscala/Function0;Lscala/collection/Seq;)V is abstract
at org.apache.spark.sql.hive.execution.command.AlterTableRenamePartitionSuite.org$apache$spark$sql$execution$command$v1$AlterTableRenamePartitionSuiteBase$$super$checkAnswer(AlterTableRenamePartitionSuite.scala)
at org.apache.spark.sql.execution.command.v1.AlterTableRenamePartitionSuiteBase.$anonfun$$init$$6(AlterTableRenamePartitionSuite.scala:60)
at org.apache.spark.sql.execution.command.v1.AlterTableRenamePartitionSuiteBase.$anonfun$$init$$6$adapted(AlterTableRenamePartitionSuite.scala:53)
```
If you have any ideas how to solve the issue, you are welcome.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758023155
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38517/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31131:
URL: https://github.com/apache/spark/pull/31131#issuecomment-758253041
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38525/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31131: [SPARK-34071][SQL][TESTS] Check stats of cached v1 tables after altering
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31131:
URL: https://github.com/apache/spark/pull/31131#discussion_r555266767
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/AlterTableAddPartitionSuite.scala
##########
@@ -46,6 +47,24 @@ trait AlterTableAddPartitionSuiteBase extends command.AlterTableAddPartitionSuit
}
}
+ private def copyPartition(t: String, from: String, to: String): String = {
Review comment:
Not sure about `from` and `to`. What do you propose here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org