You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by adrian-wang <gi...@git.apache.org> on 2014/11/06 09:49:36 UTC
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
GitHub user adrian-wang opened a pull request:
https://github.com/apache/spark/pull/3131
[SPARK-4239] [SQL] support view in HiveQl
Currently still not support view like
CREATE VIEW view3(valoo)
TBLPROPERTIES ("fear" = "factor")
AS SELECT upper(value) FROM src WHERE key=86;
because the text in metastore for this view is like
select `_c0` as `valoo` from (select upper(`src`.`value`) from `default`.`src` where ...) `view3`
while catalyst cannot resolve _c0 for this query.
For view without colname definition in parentheses, it works fine.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/adrian-wang/spark view
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3131.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3131
----
commit 88afcad1ffc0d4b74da20b5f706a835069450a57
Author: Daoyuan Wang <da...@intel.com>
Date: 2014-11-06T08:08:00Z
support view in HiveQl
commit 079290a524bf46b342ef5deb09e3bea8418d52b5
Author: Daoyuan Wang <da...@intel.com>
Date: 2014-11-06T08:42:20Z
remove useless import
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-61967571
[Test build #23001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23001/consoleFull) for PR 3131 at commit [`e46c056`](https://github.com/apache/spark/commit/e46c056ddd985859272dc0f9c57df7dba776112b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-62087390
The actual code change is `+33, -17`, so this patch should be easy to review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-61949100
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22994/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20036131
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -21,6 +21,7 @@ import java.sql.Date
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hadoop.hive.ql.Context
import org.apache.hadoop.hive.ql.lib.Node
+import org.apache.hadoop.hive.ql.metadata.Table
--- End diff --
Unused?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3131
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20051793
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -21,6 +21,7 @@ import java.sql.Date
import org.apache.hadoop.hive.conf.HiveConf
import org.apache.hadoop.hive.ql.Context
import org.apache.hadoop.hive.ql.lib.Node
+import org.apache.hadoop.hive.ql.metadata.Table
--- End diff --
line 263, declare parameter type of `createPlanForView()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-63134890
Thanks! Merged to master and 1.2
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r19989764
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
tableName: String,
alias: Option[String]): LogicalPlan = synchronized {
val (databaseName, tblName) = processDatabaseAndTableName(
- db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+ db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
val table = client.getTable(databaseName, tblName)
- val partitions: Seq[Partition] =
- if (table.isPartitioned) {
- HiveShim.getAllPartitionsOf(client, table).toSeq
- } else {
- Nil
- }
+ if (table.isView) {
+ // if the unresolved relation is from hive view
+ // parse the text into logic node.
+ HiveQl.createPlanForView(table, alias)
--- End diff --
I am wondering if we can just call the `HiveQl.createPlan()` directly here, since `alias` is not part of the `view`, and also, we may need to normalize the name of `view` or `alias`. Like below, `// Since HiveQL is case insensitive for table names we make them all lowercase.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20051950
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -428,7 +432,7 @@ private[hive] case class MetastoreRelation
}
)
- val tableDesc = HiveShim.getTableDesc(
+ lazy val tableDesc = HiveShim.getTableDesc(
--- End diff --
I tried to change this first to locate where we call `tableDesc`, it is not a necessary change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20051957
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
tableName: String,
alias: Option[String]): LogicalPlan = synchronized {
val (databaseName, tblName) = processDatabaseAndTableName(
- db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+ db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
--- End diff --
Sure, that's much clearer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-62089575
Thanks for the explanation, LGTM. +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-61974370
[Test build #23001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23001/consoleFull) for PR 3131 at commit [`e46c056`](https://github.com/apache/spark/commit/e46c056ddd985859272dc0f9c57df7dba776112b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-62262797
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23103/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20036066
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
tableName: String,
alias: Option[String]): LogicalPlan = synchronized {
val (databaseName, tblName) = processDatabaseAndTableName(
- db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+ db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
val table = client.getTable(databaseName, tblName)
- val partitions: Seq[Partition] =
- if (table.isPartitioned) {
- HiveShim.getAllPartitionsOf(client, table).toSeq
- } else {
- Nil
- }
+ if (table.isView) {
+ // if the unresolved relation is from hive view
+ // parse the text into logic node.
+ HiveQl.createPlanForView(table, alias)
--- End diff --
In general it is up to the catalog to apply `Subquery` for `aliases`, at least I think thats what we are doing elsewhere. It would be good to stay consistent unless there is some reason to change so I'm inclined to agree with @chenghao-intel.
Also I'm a little confused about where the lowercasing is happening? I agree that sort of logic belongs in the analyzer and not in the catalog. (Though maybe that comment is just stale?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-61949093
[Test build #22994 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22994/consoleFull) for PR 3131 at commit [`079290a`](https://github.com/apache/spark/commit/079290a524bf46b342ef5deb09e3bea8418d52b5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-62209309
Thanks for doing this! Since its pretty small and users have been asking for it, I'd love to squeeze this into 1.2 if possible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20036121
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -428,7 +432,7 @@ private[hive] case class MetastoreRelation
}
)
- val tableDesc = HiveShim.getTableDesc(
+ lazy val tableDesc = HiveShim.getTableDesc(
--- End diff --
Why this change?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-61974373
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23001/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-62260293
[Test build #23103 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23103/consoleFull) for PR 3131 at commit [`8a56fd6`](https://github.com/apache/spark/commit/8a56fd602dc3e6b0c91f428e40b2580387987280).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-61967157
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20051975
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
tableName: String,
alias: Option[String]): LogicalPlan = synchronized {
val (databaseName, tblName) = processDatabaseAndTableName(
- db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+ db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
val table = client.getTable(databaseName, tblName)
- val partitions: Seq[Partition] =
- if (table.isPartitioned) {
- HiveShim.getAllPartitionsOf(client, table).toSeq
- } else {
- Nil
- }
+ if (table.isView) {
+ // if the unresolved relation is from hive view
+ // parse the text into logic node.
+ HiveQl.createPlanForView(table, alias)
--- End diff --
I don't think we have to lowercase names. In resolving reference we will decide whether we should ignore case, and will get all names resolved accordingly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r20036358
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
tableName: String,
alias: Option[String]): LogicalPlan = synchronized {
val (databaseName, tblName) = processDatabaseAndTableName(
- db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+ db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
--- End diff --
This is a nit, but if we are going to reformat IMHO the correct wrapping here is:
```scala
val (databaseName, tblName) =
processDatabaseAndTableName(db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
```
I think we should always break at the highest syntactic level (i.e., in this case don't break a function from its arguments, but break the whole result from what its being assigned to)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-62262792
[Test build #23103 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23103/consoleFull) for PR 3131 at commit [`8a56fd6`](https://github.com/apache/spark/commit/8a56fd602dc3e6b0c91f428e40b2580387987280).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by adrian-wang <gi...@git.apache.org>.
Github user adrian-wang commented on a diff in the pull request:
https://github.com/apache/spark/pull/3131#discussion_r19991505
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala ---
@@ -68,19 +66,25 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
tableName: String,
alias: Option[String]): LogicalPlan = synchronized {
val (databaseName, tblName) = processDatabaseAndTableName(
- db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
+ db.getOrElse(hive.sessionState.getCurrentDatabase), tableName)
val table = client.getTable(databaseName, tblName)
- val partitions: Seq[Partition] =
- if (table.isPartitioned) {
- HiveShim.getAllPartitionsOf(client, table).toSeq
- } else {
- Nil
- }
+ if (table.isView) {
+ // if the unresolved relation is from hive view
+ // parse the text into logic node.
+ HiveQl.createPlanForView(table, alias)
--- End diff --
Since we may need some further modification in the TreeNode, I'd like to separate that from this part and do that in `createPlanForView()`. I'm not sure, we can discuss this further.
For case problem, Hive stores the table name with the original text without turn it into lowercase. And I think it is not necessary to convert that here. We can push this to reference resolving, as it already was there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4239] [SQL] support view in HiveQl
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3131#issuecomment-61944656
[Test build #22994 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22994/consoleFull) for PR 3131 at commit [`079290a`](https://github.com/apache/spark/commit/079290a524bf46b342ef5deb09e3bea8418d52b5).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org