You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by xubo245 <gi...@git.apache.org> on 2018/01/13 14:01:16 UTC
[GitHub] spark pull request #20260: [SPARK-23039][SQL] Finish TODO work in alter tabl...
GitHub user xubo245 opened a pull request:
https://github.com/apache/spark/pull/20260
[SPARK-23039][SQL] Finish TODO work in alter table set location.
## What changes were proposed in this pull request?
Finish TODO work in alter table set location.
org.apache.spark.sql.execution.command.DDLSuite#testSetLocation
// TODO(gatorsmile): fix the bug in alter table set location.
// if (isUsingHiveMetastore) {
// assert(storageFormat.properties.get("path") === expected)
// }
fix it by remove newPath = None in org.apache.spark.sql.hive.HiveExternalCatalog#restoreDataSourceTable
## How was this patch tested?
test("SPARK-23039: check path after SET LOCATION")
Wait for https://github.com/apache/spark/pull/20249
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/spark setLocationTODO
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20260.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20260
----
commit 76c1813cf6e0e0e0d085cd31dcf1633c80829eff
Author: xubo245 <60...@...>
Date: 2018-01-13T13:53:52Z
[SPARK-23039][SQL] Fix the bug in alter table set location.
TOBO work: Fix the bug in alter table set location.
org.apache.spark.sql.execution.command.DDLSuite#testSetLocation
// TODO(gatorsmile): fix the bug in alter table set location.
// if (isUsingHiveMetastore) {
// assert(storageFormat.properties.get("path") === expected)
// }
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20260
**[Test build #86103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86103/testReport)** for PR 20260 at commit [`76c1813`](https://github.com/apache/spark/commit/76c1813cf6e0e0e0d085cd31dcf1633c80829eff).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20260
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20260
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20260
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/86103/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/spark/pull/20260
I will fix the error of this PR after https://github.com/apache/spark/pull/20249#issuecomment-358720962 merged
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20260
**[Test build #86103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/86103/testReport)** for PR 20260 at commit [`76c1813`](https://github.com/apache/spark/commit/76c1813cf6e0e0e0d085cd31dcf1633c80829eff).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20260
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20260: [SPARK-23039][SQL] Finish TODO work in alter tabl...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/20260#discussion_r180181331
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -787,7 +787,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
val storageWithLocation = {
val tableLocation = getLocationFromStorageProps(table)
// We pass None as `newPath` here, to remove the path option in storage properties.
- updateLocationInStorageProps(table, newPath = None).copy(
+ table.storage.copy(
--- End diff --
In the comments above:
```
// Internally we store the table location in storage properties with key "path" for data
// source tables. Here we set the table location to `locationUri` field and filter out the
// path option in storage properties, to avoid exposing this concept externally.
```
```
// We pass None as `newPath` here, to remove the path option in storage properties.
```
The property `path` is removed intentionally.
To finish to a `TODO` comment in unit test is not a good reason to keep it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on the issue:
https://github.com/apache/spark/pull/20260
The `TODO` in comment should be removed, as I explained.
We should close this PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20260: [SPARK-23039][SQL] Finish TODO work in alter tabl...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20260#discussion_r180359425
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -787,7 +787,7 @@ private[spark] class HiveExternalCatalog(conf: SparkConf, hadoopConf: Configurat
val storageWithLocation = {
val tableLocation = getLocationFromStorageProps(table)
// We pass None as `newPath` here, to remove the path option in storage properties.
- updateLocationInStorageProps(table, newPath = None).copy(
+ table.storage.copy(
--- End diff --
So we should remove the TODO comment/work?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20260
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20260
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20260
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/20260
@xubo245 please close this if the other PR supersedes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/spark/pull/20260
This PR is different
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/spark/pull/20260
@gatorsmile Please review it. This is your TODO work.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20260: [SPARK-23039][SQL] Finish TODO work in alter table set l...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20260
cc @gengliangwang
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20260: [SPARK-23039][SQL] Finish TODO work in alter tabl...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 closed the pull request at:
https://github.com/apache/spark/pull/20260
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org