You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ravipesala <gi...@git.apache.org> on 2014/09/13 20:23:20 UTC
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
GitHub user ravipesala opened a pull request:
https://github.com/apache/spark/pull/2381
[SPARK-2594][SQL] Add CACHE TABLE <name> AS SELECT ...
This feature allows user to add cache table from the select query.
Example : ADD CACHE TABLE <tableName> AS SELECT * FROM TEST_TABLE.
Spark takes this type of SQL as command and it does eager caching.
It can be executed from SQLContext and HiveContext.
Author : ravipesala <ra...@huawei.com>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/spark Add-Cache-table-as
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2381.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2381
----
commit b803fc80efec026784b87c468b2597e5efbb6708
Author: ravipesala <ra...@huawei.com>
Date: 2014-09-11T10:23:45Z
Add CACHE TABLE <name> AS SELECT ...
This feature allows user to add cache table from the select query.
Example : ADD CACHE TABLE <tableName> AS SELECT * FROM TEST_TABLE.
Spark takes this type of SQL as command and it does eager caching.
It can be executed from SQLContext and HiveContext.
Signed-off-by: ravipesala <ra...@huawei.com>
commit 4e858d83b0020a1701ed65eac7047ee2978329db
Author: ravipesala <ra...@huawei.com>
Date: 2014-09-13T12:36:49Z
Updated parser to support add cache table command
commit 13c8e27c33e8934bbd6fb458536675e97c3d8798
Author: ravipesala <ra...@huawei.com>
Date: 2014-09-13T17:15:10Z
Updated parser to support add cache table command
commit 7459ce36775126f4c0636585c1d29f30ab35fd06
Author: ravipesala <ra...@huawei.com>
Date: 2014-09-13T17:39:28Z
Added comment
commit 6758f808d14ec7a3da0953f7720f7f5b9a4e8a85
Author: ravipesala <ra...@huawei.com>
Date: 2014-09-13T18:07:25Z
Changed style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513682
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -263,6 +262,19 @@ private[hive] object HiveQl {
}
}
+ /** Creates LogicalPlan for a given HiveQL string. */
+ def createPlan(sql: String) ={
--- End diff --
Space after `=`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513664
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala ---
@@ -119,4 +119,20 @@ class CachedTableSuite extends QueryTest {
}
assert(!TestSQLContext.isCached("testData"), "Table 'testData' should not be cached")
}
+
+ test("ADD CACHE TABLE tableName AS SELECT Star Table") {
+ TestSQLContext.sql("ADD CACHE TABLE testCacheTable AS SELECT * FROM testData")
+ TestSQLContext.sql("SELECT * FROM testCacheTable WHERE key = 1").collect()
+ TestSQLContext.uncacheTable("testCacheTable")
+ }
+
+ test("'ADD CACHE TABLE tableName AS SELECT ..'") {
+ TestSQLContext.sql("ADD CACHE TABLE testCacheTable AS SELECT * FROM testData")
+ TestSQLContext.table("testCacheTable").queryExecution.executedPlan match {
+ case _: InMemoryColumnarTableScan => // Found evidence of caching
--- End diff --
Use `isCached`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17528900
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -181,6 +182,12 @@ class SqlParser extends StandardTokenParsers with PackratParsers {
val overwrite: Boolean = o.getOrElse("") == "OVERWRITE"
InsertIntoTable(r, Map[String, Option[String]](), s, overwrite)
}
+
+ protected lazy val addCache: Parser[LogicalPlan] =
+ ADD ~ CACHE ~ TABLE ~> ident ~ AS ~ select <~ opt(";") ^^ {
--- End diff --
Thanks for your comments. Sorry for misunderstanding I updated as per the syntax ```CACHE TABLE AS SELECT ... ```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-55968221
Mind to close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-55574730
As there is a confusion in rebasing, I have created a new pull request https://github.com/apache/spark/pull/2397 rebased with master and also fixed the review comments raised here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513662
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -166,3 +166,24 @@ case class DescribeCommand(child: SparkPlan, output: Seq[Attribute])(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class CacheTableAsSelectCommand(tableName: String,plan: LogicalPlan)(
+ @transient context: SQLContext)
+ extends LeafNode with Command {
+
+ override protected[sql] lazy val sideEffectResult = {
+ context.catalog.registerTable(None, tableName, sqlContext.executePlan(plan).analyzed)
+ context.cacheTable(tableName)
+ //It does the caching eager.
+ //TODO : Does it really require to collect?
--- End diff --
I'd do a `count()` instead of a `collect()`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513651
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -166,3 +166,24 @@ case class DescribeCommand(child: SparkPlan, output: Seq[Attribute])(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class CacheTableAsSelectCommand(tableName: String,plan: LogicalPlan)(
--- End diff --
Space after `,`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-55502894
Thanks for working on this! A few minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17528957
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala ---
@@ -119,4 +119,20 @@ class CachedTableSuite extends QueryTest {
}
assert(!TestSQLContext.isCached("testData"), "Table 'testData' should not be cached")
}
+
+ test("ADD CACHE TABLE tableName AS SELECT Star Table") {
+ TestSQLContext.sql("ADD CACHE TABLE testCacheTable AS SELECT * FROM testData")
+ TestSQLContext.sql("SELECT * FROM testCacheTable WHERE key = 1").collect()
+ TestSQLContext.uncacheTable("testCacheTable")
+ }
+
+ test("'ADD CACHE TABLE tableName AS SELECT ..'") {
+ TestSQLContext.sql("ADD CACHE TABLE testCacheTable AS SELECT * FROM testData")
+ TestSQLContext.table("testCacheTable").queryExecution.executedPlan match {
+ case _: InMemoryColumnarTableScan => // Found evidence of caching
--- End diff --
OK
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17529002
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -214,6 +214,7 @@ private[hive] object HiveQl {
*/
def getAst(sql: String): ASTNode = ParseUtils.findRootNonNullToken((new ParseDriver).parse(sql))
+
--- End diff --
Sorry for trouble. I am new to git. I have removed new line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17528937
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -166,3 +166,24 @@ case class DescribeCommand(child: SparkPlan, output: Seq[Attribute])(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class CacheTableAsSelectCommand(tableName: String,plan: LogicalPlan)(
+ @transient context: SQLContext)
--- End diff --
OK. Removed passing ```sqlContext ``` manually.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-55502961
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/87/consoleFull) for PR 2381 at commit [`6758f80`](https://github.com/apache/spark/commit/6758f808d14ec7a3da0953f7720f7f5b9a4e8a85).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513670
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -214,6 +214,7 @@ private[hive] object HiveQl {
*/
def getAst(sql: String): ASTNode = ParseUtils.findRootNonNullToken((new ParseDriver).parse(sql))
+
--- End diff --
Unneeded newline.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-55502349
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17528911
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -181,6 +182,12 @@ class SqlParser extends StandardTokenParsers with PackratParsers {
val overwrite: Boolean = o.getOrElse("") == "OVERWRITE"
InsertIntoTable(r, Map[String, Option[String]](), s, overwrite)
}
+
+ protected lazy val addCache: Parser[LogicalPlan] =
--- End diff --
Updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-56020207
OK. Closing this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513680
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -243,14 +244,12 @@ private[hive] object HiveQl {
} else if (sql.trim.startsWith("!")) {
ShellCommand(sql.drop(1))
} else {
- val tree = getAst(sql)
- if (nativeCommands contains tree.getText) {
- NativeCommand(sql)
+ if (sql.trim.toLowerCase.startsWith("add cache table")) {
+ sql.trim.drop(16).split(" ").toSeq match {
+ case Seq(tableName,as, xs@_*) => CacheTableAsSelectCommand(tableName,createPlan(sql.trim.drop(16+tableName.length()+as.length()+1)))
--- End diff --
I think this line is over 100 chars: https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
You can also run `sbt/sbt scalastyle`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513645
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -181,6 +182,12 @@ class SqlParser extends StandardTokenParsers with PackratParsers {
val overwrite: Boolean = o.getOrElse("") == "OVERWRITE"
InsertIntoTable(r, Map[String, Option[String]](), s, overwrite)
}
+
+ protected lazy val addCache: Parser[LogicalPlan] =
+ ADD ~ CACHE ~ TABLE ~> ident ~ AS ~ select <~ opt(";") ^^ {
--- End diff --
Sorry for the confusion, I had intended the syntax to be `CACHE TABLE AS SELECT ...` to match `CREATE TABLE AS SELECT`. The "Add", was just about adding support to Spark SQL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17528949
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -166,3 +166,24 @@ case class DescribeCommand(child: SparkPlan, output: Seq[Attribute])(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class CacheTableAsSelectCommand(tableName: String,plan: LogicalPlan)(
+ @transient context: SQLContext)
+ extends LeafNode with Command {
+
+ override protected[sql] lazy val sideEffectResult = {
+ context.catalog.registerTable(None, tableName, sqlContext.executePlan(plan).analyzed)
+ context.cacheTable(tableName)
+ //It does the caching eager.
+ //TODO : Does it really require to collect?
--- End diff --
I have added as ```count```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17529035
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -233,7 +234,7 @@ private[hive] object HiveQl {
} else if (sql.trim.toLowerCase.startsWith("uncache table")) {
CacheCommand(sql.trim.drop(14).trim, false)
} else if (sql.trim.toLowerCase.startsWith("add jar")) {
- AddJar(sql.trim.drop(8).trim)
+ NativeCommand(sql)
--- End diff --
Sorry for trouble. I am new to git. I have faced some problems in rebase.I have reverted it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala closed the pull request at:
https://github.com/apache/spark/pull/2381
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17529073
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -243,14 +244,12 @@ private[hive] object HiveQl {
} else if (sql.trim.startsWith("!")) {
ShellCommand(sql.drop(1))
} else {
- val tree = getAst(sql)
- if (nativeCommands contains tree.getText) {
- NativeCommand(sql)
+ if (sql.trim.toLowerCase.startsWith("add cache table")) {
+ sql.trim.drop(16).split(" ").toSeq match {
+ case Seq(tableName,as, xs@_*) => CacheTableAsSelectCommand(tableName,createPlan(sql.trim.drop(16+tableName.length()+as.length()+1)))
--- End diff --
Thank you for guiding me. I have run the ```sbt/sbt scalastyle``` and updated the code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-55502990
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/87/consoleFull) for PR 2381 at commit [`6758f80`](https://github.com/apache/spark/commit/6758f808d14ec7a3da0953f7720f7f5b9a4e8a85).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class JavaSparkContext(val sc: SparkContext)`
* `class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception `
* ` class Dummy(object):`
* `case class CacheTableAsSelectCommand(tableName: String, plan: LogicalPlan) extends Command`
* `case class CacheTableAsSelectCommand(tableName: String,plan: LogicalPlan)(`
* `class JavaStreamingContext(val ssc: StreamingContext) extends Closeable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/2381#issuecomment-55552011
It seems that there are some unrelated changes in it, could you rebase with master?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17528919
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -166,3 +166,24 @@ case class DescribeCommand(child: SparkPlan, output: Seq[Attribute])(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class CacheTableAsSelectCommand(tableName: String,plan: LogicalPlan)(
--- End diff --
Ok.Updated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513672
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala ---
@@ -233,7 +234,7 @@ private[hive] object HiveQl {
} else if (sql.trim.toLowerCase.startsWith("uncache table")) {
CacheCommand(sql.trim.drop(14).trim, false)
} else if (sql.trim.toLowerCase.startsWith("add jar")) {
- AddJar(sql.trim.drop(8).trim)
+ NativeCommand(sql)
--- End diff --
Why did you change this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513661
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -166,3 +166,24 @@ case class DescribeCommand(child: SparkPlan, output: Seq[Attribute])(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class CacheTableAsSelectCommand(tableName: String,plan: LogicalPlan)(
+ @transient context: SQLContext)
--- End diff --
You do not need to manually pass a SQLContext anymore (some code in this file may be out of date). It should be accessible in all `SparkPlan`s as `sqlContext`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2594][SQL] Add CACHE TABLE AS SE...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2381#discussion_r17513648
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -181,6 +182,12 @@ class SqlParser extends StandardTokenParsers with PackratParsers {
val overwrite: Boolean = o.getOrElse("") == "OVERWRITE"
InsertIntoTable(r, Map[String, Option[String]](), s, overwrite)
}
+
+ protected lazy val addCache: Parser[LogicalPlan] =
--- End diff --
Perhaps `cacheTableAsSelect`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org