You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2017/10/12 14:08:48 UTC
[GitHub] spark pull request #19483: [SPARK-21165][SQL] FileFormatWriter should only r...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/19483
[SPARK-21165][SQL] FileFormatWriter should only rely on attributes from analyzed plan
## What changes were proposed in this pull request?
The plan output attribute ids may get changed during optimization, so `FileFormatWriter` should only use attribute ids from analyzed plan instead of optimized/physical plan, or bind error may occur.
## How was this patch tested?
new regression test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark bug2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19483.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19483
----
commit d90a0e4e8a62ffeb5742777d234db7ff3bab4272
Author: Wenchen Fan <we...@databricks.com>
Date: 2017-10-12T14:04:29Z
FileFormatWriter should only rely on attributes from analyzed plan
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19483: [SPARK-21165][SQL] FileFormatWriter should handle...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19483
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19483
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19483
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19483
**[Test build #82685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82685/testReport)** for PR 19483 at commit [`3bd5b11`](https://github.com/apache/spark/commit/3bd5b11b74459d5181275e93d11e1406cd2532a3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19483
that will be great, thanks @tejasapatil !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19483
**[Test build #82683 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82683/testReport)** for PR 19483 at commit [`d90a0e4`](https://github.com/apache/spark/commit/d90a0e4e8a62ffeb5742777d234db7ff3bab4272).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19483
thanks for the review, merging to master!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on the issue:
https://github.com/apache/spark/pull/19483
>> I'll refactor it later, to use requiredChildOrdering to do the sort.
The hive bucketing PR does that : https://github.com/apache/spark/pull/19001 I can isolate that piece and put out a PR
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19483
I'll refactor it later, to use `requiredChildOrdering` to do the sort. I just wanna make this bug fix as simple as possible.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19483
cc @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19483
It sounds like we are facing various issues because we are using the analyzed plan. Is that possible we just add an extra Project using the analyzed plan's output at the end of optimizer?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19483
**[Test build #82712 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82712/testReport)** for PR 19483 at commit [`f4a7337`](https://github.com/apache/spark/commit/f4a7337b3c4c2b58931550afc5d57902fa98ba96).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19483
**[Test build #82712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82712/testReport)** for PR 19483 at commit [`f4a7337`](https://github.com/apache/spark/commit/f4a7337b3c4c2b58931550afc5d57902fa98ba96).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19483
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82712/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19483
**[Test build #82685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82685/testReport)** for PR 19483 at commit [`3bd5b11`](https://github.com/apache/spark/commit/3bd5b11b74459d5181275e93d11e1406cd2532a3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19483
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82685/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19483
**[Test build #82683 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82683/testReport)** for PR 19483 at commit [`d90a0e4`](https://github.com/apache/spark/commit/d90a0e4e8a62ffeb5742777d234db7ff3bab4272).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should handle mismat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19483
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19483: [SPARK-21165][SQL] FileFormatWriter should only rely on ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19483
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82683/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org