You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rxin <gi...@git.apache.org> on 2016/07/01 05:56:17 UTC
[GitHub] spark pull request #14009: [SPARK-16311][SQL] Metadata refresh should work o...
GitHub user rxin opened a pull request:
https://github.com/apache/spark/pull/14009
[SPARK-16311][SQL] Metadata refresh should work on temporary views
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
Re-enabled a previously ignored test, and added a new test suite for Hive testing behavior of temporary views against MetastoreRelation.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rxin/spark SPARK-16311
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14009.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14009
----
commit cbfbbc7d27ae086805625fa41dbcbad50783fee8
Author: petermaxlee <pe...@gmail.com>
Date: 2016-06-30T04:50:37Z
[SPARK-16311][SQL] Improve metadata refresh
commit f7150345245accd0e71a351e9da9ebac9b80a520
Author: petermaxlee <pe...@gmail.com>
Date: 2016-06-30T04:53:58Z
Add test suite
commit 82f9bec79125ad3f1c4da504891a75adb5b33f2f
Author: petermaxlee <pe...@gmail.com>
Date: 2016-06-30T04:55:40Z
undo import
commit d5ea4efe8301b80dd626ae2d2bf793e4af63153c
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-07-01T05:26:52Z
Merge pull request #13989 from petermaxlee/SPARK-16311
[SPARK-16311][SQL] Improve metadata refresh
commit daac78bd5b1ed663c58821759dc0ba1a48264cea
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-07-01T05:33:37Z
Remove the refresh public API.
commit 1225cfc42e156e863842849a0b8834240c47f9ab
Author: Reynold Xin <rx...@databricks.com>
Date: 2016-07-01T05:55:47Z
Add a Hive specific test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14009
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14009
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61600/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14009
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61773/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14009
**[Test build #61773 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61773/consoleFull)** for PR 14009 at commit [`b131175`](https://github.com/apache/spark/commit/b131175b63181410437b13e2f8274e01061a2efe).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14009: [SPARK-16311][SQL] Metadata refresh should work o...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/14009#discussion_r69316671
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/MetadataCacheSuite.scala ---
@@ -77,12 +77,12 @@ class MetadataCacheSuite extends QueryTest with SharedSQLContext {
sql("select count(*) from view_refresh").first()
}
assert(e.getMessage.contains("FileNotFoundException"))
- assert(e.getMessage.contains("refresh()"))
+ assert(e.getMessage.contains("REFRESH"))
// Refresh and we should be able to read it again.
spark.catalog.refreshTable("view_refresh")
val newCount = sql("select count(*) from view_refresh").first().getLong(0)
assert(newCount > 0 && newCount < 100)
- }
+ }}
--- End diff --
This style is pretty weird...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14009: [SPARK-16311][SQL] Metadata refresh should work o...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/14009
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14009
**[Test build #61599 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61599/consoleFull)** for PR 14009 at commit [`1225cfc`](https://github.com/apache/spark/commit/1225cfc42e156e863842849a0b8834240c47f9ab).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HiveMetadataCacheSuite extends QueryTest with SQLTestUtils with TestHiveSingleton `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #14009: [SPARK-16311][SQL] Metadata refresh should work o...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14009#discussion_r69259865
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ---
@@ -462,17 +462,15 @@ class SessionCatalog(
}
}
- // TODO: It's strange that we have both refresh and invalidate here.
-
/**
* Refresh the cache entry for a metastore table, if any.
*/
- def refreshTable(name: TableIdentifier): Unit = { /* no-op */ }
-
- /**
- * Invalidate the cache entry for a metastore table, if any.
- */
- def invalidateTable(name: TableIdentifier): Unit = { /* no-op */ }
+ def refreshTable(name: TableIdentifier): Unit = {
+ // Go through temporary tables and invalidate them.
--- End diff --
In the test case of `HiveMetadataCacheSuite.scala`, users might refresh the base table by using `spark.catalog.refreshTable("view_refresh")`. Normally, they do not specify the current database name. Then, its database name is empty. Thus, this table will be treated as a temporary table. This comment might need a correction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14009
**[Test build #61773 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61773/consoleFull)** for PR 14009 at commit [`b131175`](https://github.com/apache/spark/commit/b131175b63181410437b13e2f8274e01061a2efe).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14009
**[Test build #61600 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61600/consoleFull)** for PR 14009 at commit [`0c47cca`](https://github.com/apache/spark/commit/0c47ccaa409e8e5fa60cf57b355f81a86a260792).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14009
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14009
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/61599/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the issue:
https://github.com/apache/spark/pull/14009
LGTM except for a minor styling issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/14009
Thanks - I fixed the two comments. Going to merge it in master/2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14009
LGTM except a minor comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14009
**[Test build #61599 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61599/consoleFull)** for PR 14009 at commit [`1225cfc`](https://github.com/apache/spark/commit/1225cfc42e156e863842849a0b8834240c47f9ab).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14009
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14009
Thank you for fast fix!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/14009
Hi, @rxin .
It seems this causes build failure.
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-compile-maven-hadoop-2.7/2094/console
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #14009: [SPARK-16311][SQL] Metadata refresh should work on tempo...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14009
**[Test build #61600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/61600/consoleFull)** for PR 14009 at commit [`0c47cca`](https://github.com/apache/spark/commit/0c47ccaa409e8e5fa60cf57b355f81a86a260792).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org