You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/26 06:12:16 UTC
[GitHub] [spark] wangyum opened a new pull request #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
wangyum opened a new pull request #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032
### What changes were proposed in this pull request?
This PR add two rule `RepartitionBeforeInsertDataSourceTable` and `RepartitionBeforeInsertHiveTable` to support add repartition by dynamic partition columns before insert partition table.
### Why are the changes needed?
To ease pressure on the NameNode and improve insert performance.
Before this PR. The task will throw exception and create many blocks on HDFS:
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Unit test and benchmark test:
Query | Before this PR | After this PR
-- | -- | --
CREATE TABLE t1 USING parquet PARTITIONED BY (p1, p2) AS (SELECT id, id % 1000 AS p1, id % 10000 AS p2 FROM range(5000000)) | 15 min | 1.1 min
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604909377
**[Test build #120465 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120465/testReport)** for PR 28032 at commit [`528239a`](https://github.com/apache/spark/commit/528239abac14bf5072e0ca886d857071ae8f4d64).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604853584
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604853591
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25172/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604343836
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604343849
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25125/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604855887
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604250255
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604265555
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
wangyum commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604341829
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604250255
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604852971
**[Test build #120465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120465/testReport)** for PR 28032 at commit [`528239a`](https://github.com/apache/spark/commit/528239abac14bf5072e0ca886d857071ae8f4d64).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604473172
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604909576
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604343226
**[Test build #120417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120417/testReport)** for PR 28032 at commit [`49bd334`](https://github.com/apache/spark/commit/49bd334a36cd706e2b18d54f790461e33a51dbb0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on issue #28032: [SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu commented on issue #28032: [SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-605406635
Maybe need to some special code handling REPARTITION HINT to avoid one shuffle stage?
but a little cornor case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-605013839
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120467/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604472170
**[Test build #120417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120417/testReport)** for PR 28032 at commit [`49bd334`](https://github.com/apache/spark/commit/49bd334a36cd706e2b18d54f790461e33a51dbb0).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class RepartitionBeforeInsertDataSourceTable(conf: SQLConf) extends Rule[LogicalPlan] `
* `case class RepartitionBeforeInsertHiveTable(conf: SQLConf) extends Rule[LogicalPlan] `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604265564
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120400/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604250258
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25110/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604855534
**[Test build #120467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120467/testReport)** for PR 28032 at commit [`f094e4a`](https://github.com/apache/spark/commit/f094e4a6db8c07e98a07224f54e2bdfadb3b8996).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604249797
**[Test build #120400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120400/testReport)** for PR 28032 at commit [`49bd334`](https://github.com/apache/spark/commit/49bd334a36cd706e2b18d54f790461e33a51dbb0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604909590
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120465/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604250258
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25110/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604343849
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25125/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604473193
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120417/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604909576
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604909590
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120465/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-605012774
**[Test build #120467 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120467/testReport)** for PR 28032 at commit [`f094e4a`](https://github.com/apache/spark/commit/f094e4a6db8c07e98a07224f54e2bdfadb3b8996).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604855896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25174/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604473193
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120417/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604473172
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-605013830
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604265564
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120400/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604265555
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604343226
**[Test build #120417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120417/testReport)** for PR 28032 at commit [`49bd334`](https://github.com/apache/spark/commit/49bd334a36cd706e2b18d54f790461e33a51dbb0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-605013830
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604265496
**[Test build #120400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120400/testReport)** for PR 28032 at commit [`49bd334`](https://github.com/apache/spark/commit/49bd334a36cd706e2b18d54f790461e33a51dbb0).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class RepartitionBeforeInsertDataSourceTable(conf: SQLConf) extends Rule[LogicalPlan] `
* `case class RepartitionBeforeInsertHiveTable(conf: SQLConf) extends Rule[LogicalPlan] `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604343836
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604853584
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604853591
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25172/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-605013839
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120467/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604855534
**[Test build #120467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120467/testReport)** for PR 28032 at commit [`f094e4a`](https://github.com/apache/spark/commit/f094e4a6db8c07e98a07224f54e2bdfadb3b8996).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604249797
**[Test build #120400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120400/testReport)** for PR 28032 at commit [`49bd334`](https://github.com/apache/spark/commit/49bd334a36cd706e2b18d54f790461e33a51dbb0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604855896
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25174/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28032:
[WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before
insert partition table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604855887
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL]
Repartition by dynamic partition columns before insert partition table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28032: [WIP][SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-604852971
**[Test build #120465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120465/testReport)** for PR 28032 at commit [`528239a`](https://github.com/apache/spark/commit/528239abac14bf5072e0ca886d857071ae8f4d64).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AngersZhuuuu removed a comment on issue #28032:
[SPARK-31264][SQL] Repartition by dynamic partition columns before insert
partition table
Posted by GitBox <gi...@apache.org>.
AngersZhuuuu removed a comment on issue #28032: [SPARK-31264][SQL] Repartition by dynamic partition columns before insert partition table
URL: https://github.com/apache/spark/pull/28032#issuecomment-605406635
Maybe need to some special code handling REPARTITION HINT to avoid one shuffle stage?
but a little cornor case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org