You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/16 15:52:29 UTC
[GitHub] [spark] Ngone51 opened a new pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Ngone51 opened a new pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Make static partition also follows `StoreAssignmentPolicy` when insert into table:
if `StoreAssignmentPolicy=LEGACY`, using `Cast`;
if `StoreAssignmentPolicy=ANSI | STRIC`, using `AnsiCast`;
E.g., for the table `t` created by:
```
create table t(a int, b string) using parquet partitioned by (a)
```
and insert values with `StoreAssignmentPolicy=ANSI` using:
```
insert into t partition(a='ansi') values('ansi')
```
Before this PR:
```
+----+----+
| b| a|
+----+----+
|ansi|null|
+----+----+
```
After this PR, insert will fail by:
```
java.lang.NumberFormatException: invalid input syntax for type numeric: ansi
```
(It should be better if we could use `TableOutputResolver.checkField` to fully follow `StoreAssignmentPolicy`. But since we lost the data type of static partition's value at first place, it's hard to use `TableOutputResolver.checkField`.)
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
I think we should follow `StoreAssignmentPolicy` when insert into table for any columns, including static partition.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Added new test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586748230
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118506/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589562008
**[Test build #118769 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118769/testReport)** for PR 27597 at commit [`3d10ca7`](https://github.com/apache/spark/commit/3d10ca729cb501061635093e01cab4c7161b5fcf).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589562058
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589693838
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118776/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382598164
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,17 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ // SPARK-30844: try our best to follow StoreAssignmentPolicy for static partition
+ // values but not completely follow because we can't do static type checking due to
+ // the reason that the parser has erased the type info of static partition values
+ // and converted them to string.
Review comment:
That comment has already been addressed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589564084
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23527/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586813253
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118517/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586813253
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118517/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586888256
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23314/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589117364
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23474/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589274648
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586721836
ping @cloud-fan @gengliangwang, PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589693826
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586784451
**[Test build #118517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118517/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589274648
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589117332
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382410965
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
yea let's add some comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586838036
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382065760
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
The parser erases the type info of static partition values and converts them to string, so we can't really do a compile-time type check here...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589550939
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23521/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382433518
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,17 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ // SPARK-30844: try our best to follow StoreAssignmentPolicy for static partition
+ // values but not completely follow because we can't use`DataType.canWrite` due to
Review comment:
nit: be more general like `... because we can't do static type checking due to ...`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589564084
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23527/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586784649
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586887236
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589564075
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589550932
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589534205
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586995762
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-590143469
thanks all!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586813249
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586995773
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118559/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586864376
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r379955985
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -59,7 +60,8 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
sourceAttributes: Seq[Attribute],
providedPartitions: Map[String, Option[String]],
targetAttributes: Seq[Attribute],
- targetPartitionSchema: StructType): Seq[NamedExpression] = {
+ targetPartitionSchema: StructType,
+ conf: SQLConf): Seq[NamedExpression] = {
Review comment:
Removed, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589563534
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589692499
**[Test build #118776 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118776/testReport)** for PR 27597 at commit [`3d10ca7`](https://github.com/apache/spark/commit/3d10ca729cb501061635093e01cab4c7161b5fcf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586887705
**[Test build #118559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118559/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586995773
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118559/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589550939
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23521/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589274662
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118723/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589115533
**[Test build #118723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118723/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589543648
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118762/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586784649
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586864093
**[Test build #118540 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118540/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586864382
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118540/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586784451
**[Test build #118517 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118517/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586888250
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586838040
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23295/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586722573
**[Test build #118506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118506/testReport)** for PR 27597 at commit [`9ca4a77`](https://github.com/apache/spark/commit/9ca4a778cbcb71946d6e5bcabc81b486e61e7d58).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586748228
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586837703
**[Test build #118540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118540/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589274662
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118723/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586838036
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589562065
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118769/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586888250
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586784656
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23272/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589562065
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118769/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586784656
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23272/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-590046240
Thanks! Merged to master/branch-3.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589566582
**[Test build #118776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118776/testReport)** for PR 27597 at commit [`3d10ca7`](https://github.com/apache/spark/commit/3d10ca729cb501061635093e01cab4c7161b5fcf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586748230
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118506/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586813249
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586722573
**[Test build #118506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118506/testReport)** for PR 27597 at commit [`9ca4a77`](https://github.com/apache/spark/commit/9ca4a778cbcb71946d6e5bcabc81b486e61e7d58).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586864382
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118540/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589273868
**[Test build #118723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118723/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r379956013
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala
##########
@@ -753,6 +753,19 @@ class InsertSuite extends DataSourceTest with SharedSparkSession {
}
}
+ test("SPARK-30844: static partition should also follow StoreAssignmentPolicy") {
+ withSQLConf(
+ SQLConf.STORE_ASSIGNMENT_POLICY.key -> SQLConf.StoreAssignmentPolicy.ANSI.toString) {
Review comment:
updated!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589533846
**[Test build #118762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118762/testReport)** for PR 27597 at commit [`a70c89c`](https://github.com/apache/spark/commit/a70c89c1138bd1e643fa95de8cad7c0c152ff76b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589534205
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586995762
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586722728
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23261/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r379940805
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -59,7 +60,8 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
sourceAttributes: Seq[Attribute],
providedPartitions: Map[String, Option[String]],
targetAttributes: Seq[Attribute],
- targetPartitionSchema: StructType): Seq[NamedExpression] = {
+ targetPartitionSchema: StructType,
+ conf: SQLConf): Seq[NamedExpression] = {
Review comment:
Why do we need conf here, the constructor has conf though?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589693838
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118776/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589550932
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586995033
**[Test build #118559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118559/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589115533
**[Test build #118723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118723/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589693826
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382587649
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,17 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ // SPARK-30844: try our best to follow StoreAssignmentPolicy for static partition
+ // values but not completely follow because we can't do static type checking due to
+ // the reason that the parser has erased the type info of static partition values
+ // and converted them to string.
Review comment:
Thanks for the update, @Ngone51 ! Looks fine except for the @cloud-fan comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r379940916
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala
##########
@@ -753,6 +753,19 @@ class InsertSuite extends DataSourceTest with SharedSparkSession {
}
}
+ test("SPARK-30844: static partition should also follow StoreAssignmentPolicy") {
+ withSQLConf(
+ SQLConf.STORE_ASSIGNMENT_POLICY.key -> SQLConf.StoreAssignmentPolicy.ANSI.toString) {
Review comment:
How about testing the other modes here, e.g., LEGACY and STRICT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589533846
**[Test build #118762 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118762/testReport)** for PR 27597 at commit [`a70c89c`](https://github.com/apache/spark/commit/a70c89c1138bd1e643fa95de8cad7c0c152ff76b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586838040
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23295/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382599108
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,17 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ // SPARK-30844: try our best to follow StoreAssignmentPolicy for static partition
+ // values but not completely follow because we can't do static type checking due to
+ // the reason that the parser has erased the type info of static partition values
+ // and converted them to string.
Review comment:
Oh, I missed that...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589117364
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23474/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382394314
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
Probably, better to leave some comments about why this check is placed here for making other developers understood easily.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586864376
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589534209
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23513/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589117332
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589564075
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586722728
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23261/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589543644
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586747973
**[Test build #118506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118506/testReport)** for PR 27597 at commit [`9ca4a77`](https://github.com/apache/spark/commit/9ca4a778cbcb71946d6e5bcabc81b486e61e7d58).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586888256
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23314/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589109022
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586722727
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589534209
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23513/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586836775
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r380289763
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
the major difference between ANSI and STRICT mode is: STRICT mode fails at analysis phase. Can we follow the behavior in `PreprocessTableInsertion`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586837703
**[Test build #118540 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118540/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589562058
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589550379
**[Test build #118769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118769/testReport)** for PR 27597 at commit [`3d10ca7`](https://github.com/apache/spark/commit/3d10ca729cb501061635093e01cab4c7161b5fcf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589543648
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118762/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382066581
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
Since we lost the data type of static partition values in parser, it seems hard for us to follow `canWrite` in `PreprocessTableInsertion`(because we can not figure out `writeType`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#discussion_r382432246
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
##########
@@ -104,7 +105,13 @@ case class DataSourceAnalysis(conf: SQLConf) extends Rule[LogicalPlan] with Cast
None
} else if (potentialSpecs.size == 1) {
val partValue = potentialSpecs.head._2
- Some(Alias(cast(Literal(partValue), field.dataType), field.name)())
+ conf.storeAssignmentPolicy match {
+ case StoreAssignmentPolicy.ANSI | StoreAssignmentPolicy.STRICT =>
Review comment:
Added, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586887705
**[Test build #118559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118559/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589543644
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589550379
**[Test build #118769 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118769/testReport)** for PR 27597 at commit [`3d10ca7`](https://github.com/apache/spark/commit/3d10ca729cb501061635093e01cab4c7161b5fcf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu closed pull request #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
maropu closed pull request #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586813093
**[Test build #118517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118517/testReport)** for PR 27597 at commit [`f38e056`](https://github.com/apache/spark/commit/f38e0566f3a9686828e2fe1d3f7c62cb681e4f47).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586748228
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589566582
**[Test build #118776 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118776/testReport)** for PR 27597 at commit [`3d10ca7`](https://github.com/apache/spark/commit/3d10ca729cb501061635093e01cab4c7161b5fcf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27597: [SPARK-30844][SQL]Static
partition should also follow StoreAssignmentPolicy when insert into table
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-589543557
**[Test build #118762 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118762/testReport)** for PR 27597 at commit [`a70c89c`](https://github.com/apache/spark/commit/a70c89c1138bd1e643fa95de8cad7c0c152ff76b).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27597:
[SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy
when insert into table
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27597: [SPARK-30844][SQL]Static partition should also follow StoreAssignmentPolicy when insert into table
URL: https://github.com/apache/spark/pull/27597#issuecomment-586722727
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org