You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by JiahuiJiang <gi...@git.apache.org> on 2018/03/29 17:51:18 UTC
[GitHub] spark pull request #20939: [SPARK-23823][SQL] Fix ResolveReferences
GitHub user JiahuiJiang opened a pull request:
https://github.com/apache/spark/pull/20939
[SPARK-23823][SQL] Fix ResolveReferences
## What changes were proposed in this pull request?
Fixes https://issues.apache.org/jira/browse/SPARK-23823
Introduced in apache#19585
ResolveReferences stopped doing transfromsUp after this change and Attributes sometimes lose its correct origin
## How was this patch tested?
Manually tested after this change analyze won't throw away correct Origin for columns
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JiahuiJiang/spark jj/fix-resolve-references
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20939.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20939
----
commit b4259d91679d5590c8fb343923fdf95de418687c
Author: JiahuiJiang <jj...@...>
Date: 2018-03-29T17:48:34Z
[SPARK-23823][SQL] Fix ResolveReferences
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20939
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88755/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20939
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20939
**[Test build #88755 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88755/testReport)** for PR 20939 at commit [`b5588ca`](https://github.com/apache/spark/commit/b5588ca62b645595a7a6138d0504f5e0b987d4ce).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20939
**[Test build #88755 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88755/testReport)** for PR 20939 at commit [`b5588ca`](https://github.com/apache/spark/commit/b5588ca62b645595a7a6138d0504f5e0b987d4ce).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20939
https://github.com/apache/spark/pull/20961 looks good to me. Could you close this PR? Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20939
**[Test build #88733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88733/testReport)** for PR 20939 at commit [`b5588ca`](https://github.com/apache/spark/commit/b5588ca62b645595a7a6138d0504f5e0b987d4ce).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] Fix ResolveReferences
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20939
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by JiahuiJiang <gi...@git.apache.org>.
Github user JiahuiJiang commented on the issue:
https://github.com/apache/spark/pull/20939
@gatorsmile Is #20961 crazy to do? It makes sure all the origins are kept correctly after analyzing, so future refactor won't cause similar issue. But not sure whether that behavior should be expected
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20939
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88733/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by JiahuiJiang <gi...@git.apache.org>.
Github user JiahuiJiang commented on the issue:
https://github.com/apache/spark/pull/20939
Yes, for any SQL queries with columns inside.
e.g. "create table `/targetTable` as (\nselect col1 from `/path/to/table1`\n)", after parsing the query, I got a QueryPlan where the "col1" project's origin is (2, 7). But after running analyzer.analyze on that query plan, the origin becomes (2, 0). Tested after this change it retains as (2, 7)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20939
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20939
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20939
@JiahuiJiang Why do you need to check the value of origin? This is being used for error reporting only.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20939
**[Test build #88733 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88733/testReport)** for PR 20939 at commit [`b5588ca`](https://github.com/apache/spark/commit/b5588ca62b645595a7a6138d0504f5e0b987d4ce).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by JiahuiJiang <gi...@git.apache.org>.
Github user JiahuiJiang commented on the issue:
https://github.com/apache/spark/pull/20939
@gatorsmile Seems the tests are still flaking :/
And @robertzk proposed this fix https://github.com/JiahuiJiang/spark/pull/2 instead, where we keep the origin directly inside transformExpression, so that all the mapChildren and mapExpressions automatically have their origins kept. Is it better to take that change instead?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20939
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20939: [SPARK-23823][SQL] ResolveReferences should prese...
Posted by JiahuiJiang <gi...@git.apache.org>.
Github user JiahuiJiang closed the pull request at:
https://github.com/apache/spark/pull/20939
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] Fix ResolveReferences
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20939
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by JiahuiJiang <gi...@git.apache.org>.
Github user JiahuiJiang commented on the issue:
https://github.com/apache/spark/pull/20939
We are implementing a [language server](https://microsoft.github.io/language-server-protocol/) that needs to understand where the column locations, so we are depending on some of these internal APIs
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20939: [SPARK-23823][SQL] ResolveReferences should preserve tre...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20939
@JiahuiJiang Did you hit any error?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org