You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/12 18:06:52 UTC
[GitHub] [spark] MaxGekk opened a new pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
MaxGekk opened a new pull request #31157:
URL: https://github.com/apache/spark/pull/31157
### What changes were proposed in this pull request?
Fix an issue in `ALTER TABLE .. ADD PARTITION` which happens when:
- A table doesn't have stats
- `spark.sql.statistics.size.autoUpdate.enabled` is `true`
In that case, `ALTER TABLE .. ADD PARTITION` does not update table stats automatically.
### Why are the changes needed?
The changes fix the issue demonstrated by the example:
```sql
spark-sql> create table tbl (col0 int, part int) partitioned by (part);
spark-sql> insert into tbl partition (part = 0) select 0;
spark-sql> set spark.sql.statistics.size.autoUpdate.enabled=true;
spark-sql> alter table tbl add partition (part = 1);
```
the `add partition` command should update table stats but it does not. There are no stats in the output of:
```
spark-sql> describe table extended tbl;
```
### Does this PR introduce _any_ user-facing change?
Yes. After the changes, `ALTER TABLE .. ADD PARTITION` updates stats even when a table doesn't have stats before the command:
```sql
spark-sql> alter table tbl add partition (part = 1);
spark-sql> describe table extended tbl;
col0 int NULL
part int NULL
# Partition Information
# col_name data_type comment
part int NULL
# Detailed Table Information
...
Statistics 2 bytes
```
### How was this patch tested?
By running new UT and existing test suites:
```
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *.StatisticsSuite"
```
Authored-by: Max Gekk <ma...@gmail.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit 6c047958f9fcf4cac848695915deea289c65ddc1)
Signed-off-by: Max Gekk <ma...@gmail.com>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-759103543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133979/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-759102268
**[Test build #133979 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133979/testReport)** for PR 31157 at commit [`960f158`](https://github.com/apache/spark/commit/960f158c6e8ff680076787025f4a23192cf51039).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-758885693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38567/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-758881356
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38567/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-758878860
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38567/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-758857488
**[Test build #133979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133979/testReport)** for PR 31157 at commit [`960f158`](https://github.com/apache/spark/commit/960f158c6e8ff680076787025f4a23192cf51039).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-758885693
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38567/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-759212858
thanks, merging to 3.1!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-758857488
**[Test build #133979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133979/testReport)** for PR 31157 at commit [`960f158`](https://github.com/apache/spark/commit/960f158c6e8ff680076787025f4a23192cf51039).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31157:
URL: https://github.com/apache/spark/pull/31157
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31157: [SPARK-34084][SQL][3.1] Fix auto updating of table stats in `ALTER TABLE .. ADD PARTITION`
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31157:
URL: https://github.com/apache/spark/pull/31157#issuecomment-759103543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133979/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org