You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by OopsOutOfMemory <gi...@git.apache.org> on 2015/01/07 18:06:50 UTC
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
GitHub user OopsOutOfMemory opened a pull request:
https://github.com/apache/spark/pull/3935
[SPARK-5135][SQL] Add support for describe [extended] table to DDL in SQLContext
RT.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/OopsOutOfMemory/spark describe
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3935.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3935
----
commit a4459e20e6a3370858407881b07cd090ae997f03
Author: OopsOutOfMemory <vi...@126.com>
Date: 2015-01-07T11:17:32Z
initial
commit c9f3ed8fb04308e30ede41d9871e557b01ffde90
Author: OopsOutOfMemory <vi...@126.com>
Date: 2015-01-07T11:25:11Z
initial
commit 0b67c1be819b74d301e1f5c121d1c8830ae80d2f
Author: OopsOutOfMemory <vi...@126.com>
Date: 2015-01-07T11:35:14Z
initial
commit 11535dc3639a7f58136f41012f4a22951c0164e3
Author: OopsOutOfMemory <vi...@126.com>
Date: 2015-01-07T16:47:01Z
add support describe table
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69162373
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25216/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69294092
[Test build #25296 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25296/consoleFull) for PR 3935 at commit [`b5caa6e`](https://github.com/apache/spark/commit/b5caa6e6d3e4c3c83728c968228b4c4c6a54baab).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69191053
test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22640443
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
- CommandStrategy ::
- DataSourceStrategy ::
- TakeOrdered ::
- HashAggregation ::
- LeftSemiJoin ::
- HashJoin ::
- InMemoryScans ::
- ParquetOperations ::
- BasicOperators ::
- CartesianProduct ::
- BroadcastNestedLoopJoin :: Nil)
+ extraStrategies ++ Seq(
+ CommandStrategy,
+ DDLStrategy(self),
--- End diff --
Probably we need to move `DDLStrategy` ahead of `CommandStrategy`, as @scwf described.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70776973
Yea we can close this one. Will merge that one when tests pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69190736
[Test build #25226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25226/consoleFull) for PR 3935 at commit [`47f5593`](https://github.com/apache/spark/commit/47f5593cb0d585e74cab1e3b10e9ed812e66b751).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22640643
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -326,4 +327,17 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
case _ => Nil
}
}
+
+ case class DDLStrategy(context: SQLContext) extends Strategy {
+ def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
+ case describe: logical.DescribeCommand =>
+ val resolvedTable = context.executePlan(describe.table).analyzed
+ resolvedTable match {
+ case LogicalRelation(relation) =>
+ Seq(ExecutedCommand(DDLDescribeCommand(relation, describe.output, describe.isExtended)))
+ case o: logical.DescribeCommand =>
+ ExecutedCommand(DescribeCommand(planLater(o), describe.output)) :: Nil
--- End diff --
I think @scwf is right, @OopsOutOfMemory can you double check with that by removing the `DDLStrategy`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70285046
[Test build #25665 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25665/consoleFull) for PR 3935 at commit [`d1689e2`](https://github.com/apache/spark/commit/d1689e2fdfed67decabe6c696a3d7f051138a7ad).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69485332
[Test build #25373 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25373/consoleFull) for PR 3935 at commit [`b25d32a`](https://github.com/apache/spark/commit/b25d32a93286c900f6bfc83b79fe7c8a81e1f08c).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69485494
[Test build #25374 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25374/consoleFull) for PR 3935 at commit [`7d0d1c9`](https://github.com/apache/spark/commit/7d0d1c9b1f609c02a689e362d21762a548a716f1).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69054145
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69136217
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25198/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70228217
[Test build #25651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25651/consoleFull) for PR 3935 at commit [`9efdf35`](https://github.com/apache/spark/commit/9efdf353af71b8b3a15f5a0e837e580237cfcd50).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22640124
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -181,3 +183,25 @@ case class DescribeCommand(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+ relation: BaseRelation,
+ override val output: Seq[Attribute],
+ isExtended: Boolean) extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext) = {
+ val rows = new ArrayBuffer[Row]()
+ rows += Row("# col_name", "data_type", "comment")
--- End diff --
`"# col_name", "data_type", "comment"` is the output field name, we'd better not take that as part of the output.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70198837
BTW the data type refactoring PR I was referring to was https://github.com/apache/spark/pull/3958
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22629657
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
--- End diff --
Seems a bug of error reporting, anyway, the root reason is it failed in the code style checking.
`/home/jenkins/workspace/SparkPullRequestBuilder/sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala:198:21: Insert a space after the start of the comment`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70243136
[Test build #25654 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25654/consoleFull) for PR 3935 at commit [`9d22708`](https://github.com/apache/spark/commit/9d22708ee3360bc0f9f06a5428f01ff3c7acb48d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70237708
[Test build #25655 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25655/consoleFull) for PR 3935 at commit [`5b7ae19`](https://github.com/apache/spark/commit/5b7ae19dfdc4410f1018193e0b1701abf799c439).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22762487
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala ---
@@ -77,6 +78,16 @@ private[sql] class DDLParser extends StandardTokenParsers with PackratParsers wi
CreateTableUsing(tableName, provider, opts)
}
+ /*
--- End diff --
Nit: this should be `/**`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69191464
[Test build #25227 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25227/consoleFull) for PR 3935 at commit [`f5821ae`](https://github.com/apache/spark/commit/f5821aefa29ab8c670cd513e223f73ba5a84b2ca).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69220158
@OopsOutOfMemory, test failed for hive describe cmd, this is because now hive describe will go into ddlParser not hiveQL
```
new SchemaRDD(this, ddlParser(sqlText).getOrElse(HiveQl.parseSql(sqlText))
```
A quick fix can be
```
new SchemaRDD(this, HiveQl.parseSql(sqlText).getOrElse(ddlParser(sqlText))
```
Or to make sure ddl describe has the same behavior with hive describe.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69111603
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25179/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69204168
[Test build #25231 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25231/consoleFull) for PR 3935 at commit [`9d6861a`](https://github.com/apache/spark/commit/9d6861a88c826e1f8d4f11bb41a7ae2276f54d5f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69294135
[Test build #25296 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25296/consoleFull) for PR 3935 at commit [`b5caa6e`](https://github.com/apache/spark/commit/b5caa6e6d3e4c3c83728c968228b4c4c6a54baab).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69686968
[Test build #25447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25447/consoleFull) for PR 3935 at commit [`2d159e5`](https://github.com/apache/spark/commit/2d159e5e32f31bd1a4dfaf05388eab4faa3ea1b4).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22657252
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -181,3 +183,32 @@ case class DescribeCommand(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+ dbName: Option[String],
+ tableName: String, isExtended: Boolean) extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext) = {
+
+ val tblRelation = UnresolvedRelation(dbName,tableName)
+ val logicalRelation = sqlContext.executePlan(tblRelation).analyzed
+ val rows = new ArrayBuffer[Row]()
+ rows ++= logicalRelation.schema.fields.map{field =>
+ Row(field.name, field.dataType.toString, null)}
+
+ /*
+ * TODO if future support partition table, add header below:
+ * # Partition Information
+ * # col_name data_type comment
+ */
+ if (isExtended) { // TODO describe extended table
+ rows += Row("# extended", null,null)
--- End diff --
space after `,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22640526
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -181,3 +183,25 @@ case class DescribeCommand(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+ relation: BaseRelation,
+ override val output: Seq[Attribute],
+ isExtended: Boolean) extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext) = {
+ val rows = new ArrayBuffer[Row]()
+ rows += Row("# col_name", "data_type", "comment")
--- End diff --
@chenghao-intel
ok, I will remove it.
If future support `partition table` here, I will add it like hive does:
```
# Partition Information
# col_name data_type comment
patition_col_name string None
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70772639
Thanks @rxin
Should I close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22657121
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -17,6 +17,7 @@
package org.apache.spark.sql.execution
+import org.apache.spark.sql.sources.LogicalRelation
--- End diff --
probably an unnecessary change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22763208
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/DDLTestSuit.scala ---
@@ -0,0 +1,74 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.sql.sources
+import org.apache.spark.sql._
+
+
+class DDLScanSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleDDLScan(parameters("from").toInt, parameters("TO").toInt)(sqlContext)
+ }
+}
+
+case class SimpleDDLScan(from: Int, to: Int)(@transient val sqlContext: SQLContext)
+ extends TableScan {
+
+ override def schema =
+ StructType(Seq(StructField("name", StringType, nullable = false),
+ StructField("age", IntegerType, nullable = false)))
+
+ override def buildScan() = sqlContext.sparkContext.parallelize(from to to).
+ map(e => Row(s"people$e",e*2))
+}
+
+class DDLTestSuit extends DataSourceTest {
+ import caseInsensisitiveContext._
+
+ before {
+ sql(
+ """
+ |CREATE TEMPORARY TABLE ddlPeople
+ |USING org.apache.spark.sql.sources.DDLScanSource
+ |OPTIONS (
+ | From '1',
+ | To '10'
+ |)
+ """.stripMargin)
+ }
+
+ sqlTest(
+ "SELECT * FROM ddlPeople",
+ (1 to 10).map(e => Row(s"people$e",e*2)).toSeq)
+
+ sqlTest(
+ "describe ddlPeople",
+ Seq(
+ Row("name", "StringType", null),
+ Row("age", "IntegerType", null)
+ ))
+
+ sqlTest(
+ "describe extended ddlPeople",
--- End diff --
@marmbrus
For 'extended' feature, I think we can display the user passed parameters like sth in `OPTIONS`, what do you think ? But seems there is no interface we can get the user passed parameters in the External Datasource API.
/cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69121415
That's a nice feature in general, I agree with @marmbrus , we do need a test suite for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69215154
[Test build #25232 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25232/consoleFull) for PR 3935 at commit [`a4489c1`](https://github.com/apache/spark/commit/a4489c1e14854bfcd717a28f713500f4c26300ac).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69190870
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25226/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22628909
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
--- End diff --
Nit: `a :: b :: c :: Nil` is exact the same with `Seq(a, b, c)`. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69690973
[Test build #25455 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25455/consoleFull) for PR 3935 at commit [`03a1b6c`](https://github.com/apache/spark/commit/03a1b6cfa34b0d94ca75d7ce56ca72377190ef57).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70220923
Not sure - do you have a backup? Maybe just take a diff and apply the diff on master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69155022
[Test build #25216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25216/consoleFull) for PR 3935 at commit [`fbe6f6d`](https://github.com/apache/spark/commit/fbe6f6d1edcf2ce7f3b5b18ef3dc58bfc96b6b38).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69198077
[Test build #25227 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25227/consoleFull) for PR 3935 at commit [`f5821ae`](https://github.com/apache/spark/commit/f5821aefa29ab8c670cd513e223f73ba5a84b2ca).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22764434
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/DDLTestSuit.scala ---
@@ -0,0 +1,74 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.sql.sources
+import org.apache.spark.sql._
+
+
+class DDLScanSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleDDLScan(parameters("from").toInt, parameters("TO").toInt)(sqlContext)
+ }
+}
+
+case class SimpleDDLScan(from: Int, to: Int)(@transient val sqlContext: SQLContext)
+ extends TableScan {
+
+ override def schema =
+ StructType(Seq(StructField("name", StringType, nullable = false),
+ StructField("age", IntegerType, nullable = false)))
+
+ override def buildScan() = sqlContext.sparkContext.parallelize(from to to).
+ map(e => Row(s"people$e",e*2))
+}
+
+class DDLTestSuit extends DataSourceTest {
+ import caseInsensisitiveContext._
+
+ before {
+ sql(
+ """
+ |CREATE TEMPORARY TABLE ddlPeople
+ |USING org.apache.spark.sql.sources.DDLScanSource
+ |OPTIONS (
+ | From '1',
+ | To '10'
+ |)
+ """.stripMargin)
+ }
+
+ sqlTest(
+ "SELECT * FROM ddlPeople",
+ (1 to 10).map(e => Row(s"people$e",e*2)).toSeq)
+
+ sqlTest(
+ "describe ddlPeople",
+ Seq(
+ Row("name", "StringType", null),
--- End diff --
`simpleString ` fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69294674
[Test build #25297 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25297/consoleFull) for PR 3935 at commit [`80becf3`](https://github.com/apache/spark/commit/80becf3045a743097f1922fdc4a7308bdd39c2a2).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69195423
[Test build #25230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25230/consoleFull) for PR 3935 at commit [`04c0b6f`](https://github.com/apache/spark/commit/04c0b6f728aa8081189ab996467956b043d14a6a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69132307
@chenghao-intel @marmbrus
Added test cases.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22629136
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
--- End diff --
yeah, I just make this same as hive context way.
@chenghao-intel
This test failed with `Refer to this link for build results (access rights to CI server needed): `
no rights to CL server ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22656959
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala ---
@@ -77,6 +80,15 @@ private[sql] class DDLParser extends StandardTokenParsers with PackratParsers wi
CreateTableUsing(tableName, provider, opts)
}
+ /*
+ * describe [extended] table jsonTable
+ */
+ protected lazy val describeTable: Parser[LogicalPlan] =
+ (DESCRIBE ~> opt(EXTENDED)) ~ (ident <~ ".").? ~ ident ^^ {
+ case e ~ db ~ tbl =>
+ DDLDescribeCommand(db,tbl, e.nonEmpty)
--- End diff --
Nit: space after `db,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69111169
you'll also need to add test cases
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69294136
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25296/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22658550
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
--- End diff --
I think it's ok, it can at least save some space since `::` is 2 chars, `,` is 1 char.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70220078
@rxin The rebase may have some problems, cause changed 278 files ? how do I revert it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69190858
[Test build #25226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25226/consoleFull) for PR 3935 at commit [`47f5593`](https://github.com/apache/spark/commit/47f5593cb0d585e74cab1e3b10e9ed812e66b751).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(dbName: Option[String], tableName: String, isExtended: Boolean) extends RunnableCommand `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r23269711
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -70,8 +71,15 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) {
if (conf.dialect == "sql") {
super.sql(sqlText)
} else if (conf.dialect == "hiveql") {
- new SchemaRDD(this, ddlParser(sqlText).getOrElse(HiveQl.parseSql(sqlText)))
- } else {
+ val ddlPlan = ddlParser(sqlText)
+ val basicPlan = try {
+ HiveQl.parseSql(sqlText)
+ }catch {
--- End diff --
add a space here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r23059758
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
--- End diff --
As @chenghao-intel said, do you mind not changing this part? It would be great if the PR focuses on what it does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69687277
[Test build #25447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25447/consoleFull) for PR 3935 at commit [`2d159e5`](https://github.com/apache/spark/commit/2d159e5e32f31bd1a4dfaf05388eab4faa3ea1b4).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69291456
@scwf Thanks for point it out. Updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69208329
[Test build #25232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25232/consoleFull) for PR 3935 at commit [`a4489c1`](https://github.com/apache/spark/commit/a4489c1e14854bfcd717a28f713500f4c26300ac).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69195054
This look much better now, except some of the code style issues, the other LGTM.
PS: https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69111498
[Test build #25179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25179/consoleFull) for PR 3935 at commit [`1c02744`](https://github.com/apache/spark/commit/1c0274425b1d911c217999c304277d3137f7e390).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70244543
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25655/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22629273
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -181,3 +182,23 @@ case class DescribeCommand(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+ relation: BaseRelation,
+ override val output: Seq[Attribute],
+ isExtended: Boolean) extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext) = {
+ var rows = Row("# col_name", "data_type", "comment") +:
--- End diff --
I guess using `scala.collection.mutable.ArrayBuffer` instead probably more clean.
```
val rows = new ArrayBuffer[Row]()
rows += row
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69197155
[Test build #25231 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25231/consoleFull) for PR 3935 at commit [`9d6861a`](https://github.com/apache/spark/commit/9d6861a88c826e1f8d4f11bb41a7ae2276f54d5f).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69132309
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70243143
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25654/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70217298
[Test build #25639 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25639/consoleFull) for PR 3935 at commit [`521bbd7`](https://github.com/apache/spark/commit/521bbd73b83223534a1397aa936a6bab554da7d7).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69204179
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25231/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69689308
[Test build #25452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25452/consoleFull) for PR 3935 at commit [`a2cbf6b`](https://github.com/apache/spark/commit/a2cbf6b18bccf32caf2b49796668b6e820c9ce85).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70232782
[Test build #25654 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25654/consoleFull) for PR 3935 at commit [`9d22708`](https://github.com/apache/spark/commit/9d22708ee3360bc0f9f06a5428f01ff3c7acb48d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70244538
[Test build #25655 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25655/consoleFull) for PR 3935 at commit [`5b7ae19`](https://github.com/apache/spark/commit/5b7ae19dfdc4410f1018193e0b1701abf799c439).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69859023
ping /cc @rxin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22762496
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/DDLTestSuit.scala ---
@@ -0,0 +1,74 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.sql.sources
+import org.apache.spark.sql._
+
+
+class DDLScanSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleDDLScan(parameters("from").toInt, parameters("TO").toInt)(sqlContext)
+ }
+}
+
+case class SimpleDDLScan(from: Int, to: Int)(@transient val sqlContext: SQLContext)
+ extends TableScan {
+
+ override def schema =
+ StructType(Seq(StructField("name", StringType, nullable = false),
+ StructField("age", IntegerType, nullable = false)))
+
+ override def buildScan() = sqlContext.sparkContext.parallelize(from to to).
+ map(e => Row(s"people$e",e*2))
+}
+
+class DDLTestSuit extends DataSourceTest {
+ import caseInsensisitiveContext._
+
+ before {
+ sql(
+ """
+ |CREATE TEMPORARY TABLE ddlPeople
+ |USING org.apache.spark.sql.sources.DDLScanSource
+ |OPTIONS (
+ | From '1',
+ | To '10'
+ |)
+ """.stripMargin)
+ }
+
+ sqlTest(
+ "SELECT * FROM ddlPeople",
+ (1 to 10).map(e => Row(s"people$e",e*2)).toSeq)
+
+ sqlTest(
+ "describe ddlPeople",
+ Seq(
+ Row("name", "StringType", null),
+ Row("age", "IntegerType", null)
+ ))
+
+ sqlTest(
+ "describe extended ddlPeople",
--- End diff --
I'd probably leave the extended feature out for now unless it is actually going to do something.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69484328
[Test build #25374 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25374/consoleFull) for PR 3935 at commit [`7d0d1c9`](https://github.com/apache/spark/commit/7d0d1c9b1f609c02a689e362d21762a548a716f1).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69199787
[Test build #25229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25229/consoleFull) for PR 3935 at commit [`aa34164`](https://github.com/apache/spark/commit/aa3416487629fa35d2d85a1c3544e905f5df1015).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70199626
Thanks. @rxin I'wll make this up-to-date.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69198091
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25227/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69690119
[Test build #25452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25452/consoleFull) for PR 3935 at commit [`a2cbf6b`](https://github.com/apache/spark/commit/a2cbf6b18bccf32caf2b49796668b6e820c9ce85).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70230293
@OopsOutOfMemory do like this:
git checkout describe
git pull apachesparkrepo
git fetch apachesparkrepo master:newdescribe
git diff newdescribe > describe.patch
git checkout newdescribe
git apply describe.patch
git push --force your-github-repo newdescribe:describe
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r23059763
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala ---
@@ -17,6 +17,11 @@
package org.apache.spark.sql.sources
+import org.apache.spark.Logging
--- End diff --
imports out of order here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22657039
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -181,3 +183,32 @@ case class DescribeCommand(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+ dbName: Option[String],
+ tableName: String, isExtended: Boolean) extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext) = {
+
--- End diff --
Nit: Remove the empty line.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70273405
[Test build #25665 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25665/consoleFull) for PR 3935 at commit [`d1689e2`](https://github.com/apache/spark/commit/d1689e2fdfed67decabe6c696a3d7f051138a7ad).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69690124
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25452/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69454126
[Test build #25359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25359/consoleFull) for PR 3935 at commit [`e195667`](https://github.com/apache/spark/commit/e1956671d3c958ec248eae6dcb1984f12cb3158d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r23269744
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -70,8 +71,15 @@ class HiveContext(sc: SparkContext) extends SQLContext(sc) {
if (conf.dialect == "sql") {
super.sql(sqlText)
} else if (conf.dialect == "hiveql") {
- new SchemaRDD(this, ddlParser(sqlText).getOrElse(HiveQl.parseSql(sqlText)))
- } else {
+ val ddlPlan = ddlParser(sqlText)
+ val basicPlan = try {
+ HiveQl.parseSql(sqlText)
+ }catch {
--- End diff --
actually it's ok i will fix it myself
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69202386
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25230/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69303311
retest it please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69299944
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25301/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22641890
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
- CommandStrategy ::
- DataSourceStrategy ::
- TakeOrdered ::
- HashAggregation ::
- LeftSemiJoin ::
- HashJoin ::
- InMemoryScans ::
- ParquetOperations ::
- BasicOperators ::
- CartesianProduct ::
- BroadcastNestedLoopJoin :: Nil)
+ extraStrategies ++ Seq(
+ CommandStrategy,
+ DDLStrategy(self),
--- End diff --
Sorry, what does `move DDLStrategy ahead of CommandStrategy` means: change the order to ?
```
DDLStrategy(self),
CommandStrategy,
```
scfw's idea is to `remove` this `DDLStrategy` and use `execution.DDLDescribeCommand` instead but not `move`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22629532
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -326,4 +327,17 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
case _ => Nil
}
}
+
+ case class DDLStrategy(context: SQLContext) extends Strategy {
+ def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
+ case describe: logical.DescribeCommand =>
+ val resolvedTable = context.executePlan(describe.table).analyzed
+ resolvedTable match {
+ case LogicalRelation(relation) =>
+ Seq(ExecutedCommand(DDLDescribeCommand(relation, describe.output, describe.isExtended)))
+ case o: logical.DescribeCommand =>
+ ExecutedCommand(DescribeCommand(planLater(o), describe.output)) :: Nil
--- End diff --
I am a little confusing here, does that mean we will support the command like `describe describe [extended] tbl`? Sorry if I am wrong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory closed the pull request at:
https://github.com/apache/spark/pull/3935
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69694877
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25455/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22656997
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/DDLTestSuit.scala ---
@@ -0,0 +1,75 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.sql.sources
+import org.apache.spark.sql._
+
+
+class DDLScanSource extends RelationProvider {
+ override def createRelation(
--- End diff --
Indent should be 2 spaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69687278
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25447/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69297004
[Test build #25297 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25297/consoleFull) for PR 3935 at commit [`80becf3`](https://github.com/apache/spark/commit/80becf3045a743097f1922fdc4a7308bdd39c2a2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69191016
@scwf @chenghao-intel
I've changed the implementation to RunnableCommand and removed `DDLStrategy`.
Could u review it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69360125
@marmbrus
I think it's ok to merge, could u review this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22636181
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -326,4 +327,17 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
case _ => Nil
}
}
+
+ case class DDLStrategy(context: SQLContext) extends Strategy {
+ def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
+ case describe: logical.DescribeCommand =>
+ val resolvedTable = context.executePlan(describe.table).analyzed
+ resolvedTable match {
+ case LogicalRelation(relation) =>
+ Seq(ExecutedCommand(DDLDescribeCommand(relation, describe.output, describe.isExtended)))
+ case o: logical.DescribeCommand =>
+ ExecutedCommand(DescribeCommand(planLater(o), describe.output)) :: Nil
--- End diff --
Maybe we do not need this strategy. In ddl.scala we parse a runnable ```DDLDescribeCommand``` not logical.DescribeCommand, then it will follow commandstrategy
```
case object CommandStrategy extends Strategy {
def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
case r: RunnableCommand => ExecutedCommand(r) :: Nil
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69111151
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69694874
[Test build #25455 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25455/consoleFull) for PR 3935 at commit [`03a1b6c`](https://github.com/apache/spark/commit/03a1b6cfa34b0d94ca75d7ce56ca72377190ef57).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r23059970
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
--- End diff --
@chenghao-intel @rxin
ok, got it. I will change this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69195441
@chenghao-intel
Thank you very much : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69485497
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25374/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69474872
/cc @rxin
Here is another public API we should consider standardizing for 1.3. Do we want to have a unified `DESCRIBE` that works for both SQL and Hive tables? Is it important to match Hive's output?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69192999
[Test build #25229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25229/consoleFull) for PR 3935 at commit [`aa34164`](https://github.com/apache/spark/commit/aa3416487629fa35d2d85a1c3544e905f5df1015).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69297008
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25297/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69485874
Updated. Could u review it ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22629310
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -181,3 +182,23 @@ case class DescribeCommand(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+ relation: BaseRelation,
+ override val output: Seq[Attribute],
+ isExtended: Boolean) extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext) = {
+ var rows = Row("# col_name", "data_type", "comment") +:
+ relation.schema.fields.map(field => Row(field.name, field.dataType.toString, null))
+
+ if(isExtended) { //TODO describe extended table
--- End diff --
Nit: space after `if`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69111601
[Test build #25179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25179/consoleFull) for PR 3935 at commit [`1c02744`](https://github.com/apache/spark/commit/1c0274425b1d911c217999c304277d3137f7e390).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class DDLStrategy(context: SQLContext) extends Strategy `
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69308515
[Test build #25312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25312/consoleFull) for PR 3935 at commit [`8f25bd7`](https://github.com/apache/spark/commit/8f25bd7734c35349057c0b891517390b6ea902ad).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69215161
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25232/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70285060
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25665/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70229216
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25651/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70764410
Thanks. I resolved the conflict and pushed a PR https://github.com/apache/spark/pull/4127
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69136211
[Test build #25198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25198/consoleFull) for PR 3935 at commit [`d0fe2d6`](https://github.com/apache/spark/commit/d0fe2d6404cbc27abca71ce9b42c5300577f4759).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class DDLStrategy(context: SQLContext) extends Strategy `
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70229207
[Test build #25651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25651/consoleFull) for PR 3935 at commit [`9efdf35`](https://github.com/apache/spark/commit/9efdf353af71b8b3a15f5a0e837e580237cfcd50).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69162366
[Test build #25216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25216/consoleFull) for PR 3935 at commit [`fbe6f6d`](https://github.com/apache/spark/commit/fbe6f6d1edcf2ce7f3b5b18ef3dc58bfc96b6b38).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70198790
Hi @OopsOutOfMemory,
I was holding off this one due to the data type refactoring. Now the data type refactoring is in, do you mind bringing it up to date? This is definitely something we would like to support.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69485334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25373/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69455901
[Test build #25359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25359/consoleFull) for PR 3935 at commit [`e195667`](https://github.com/apache/spark/commit/e1956671d3c958ec248eae6dcb1984f12cb3158d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22629203
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala ---
@@ -77,6 +80,16 @@ private[sql] class DDLParser extends StandardTokenParsers with PackratParsers wi
CreateTableUsing(tableName, provider, opts)
}
+ /*
+ * describe [extended] table jsonTable
+ */
+ protected lazy val describeTable: Parser[LogicalPlan] =
+ (DESCRIBE ~> opt(EXTENDED)) ~ (ident <~ ".").? ~ ident ^^ {
+ case e ~ db ~ tbl =>
+ val table = UnresolvedRelation(db,tbl)
+ DescribeCommand(table, !e.isEmpty)
--- End diff --
Nit: `e.isEmpty` == `e.nonEmpty`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69202377
[Test build #25230 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25230/consoleFull) for PR 3935 at commit [`04c0b6f`](https://github.com/apache/spark/commit/04c0b6f728aa8081189ab996467956b043d14a6a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22657062
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -181,3 +183,32 @@ case class DescribeCommand(
child.output.map(field => Row(field.name, field.dataType.toString, null))
}
}
+
+/**
+ * :: DeveloperApi ::
+ */
+@DeveloperApi
+case class DDLDescribeCommand(
+ dbName: Option[String],
+ tableName: String, isExtended: Boolean) extends RunnableCommand {
+
+ override def run(sqlContext: SQLContext) = {
+
+ val tblRelation = UnresolvedRelation(dbName,tableName)
--- End diff --
Nit: space after `,`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22657209
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala ---
@@ -328,18 +328,19 @@ class SQLContext(@transient val sparkContext: SparkContext)
def numPartitions = self.numShufflePartitions
def strategies: Seq[Strategy] =
- extraStrategies ++ (
--- End diff --
Can we revert `SQLContext` in this PR? Since the change is equivalent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69297563
[Test build #25301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25301/consoleFull) for PR 3935 at commit [`4a24144`](https://github.com/apache/spark/commit/4a24144d774f61c9d539dfb38baae369d3525aa0).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70237945
Thanks @scwf @rxin, conflicts resolved cleanly, this now up-to-date.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69056638
e.g.
```
import org.apache.spark.sql.SQLContext
val sqlContext = new SQLContext(sc)
import sqlContext._
val jsonDDL = s"""
|CREATE TEMPORARY TABLE jsonTable
|USING org.apache.spark.sql.json
|OPTIONS (
| path 'file:///Users/shengli/git_repos/spark/examples/src/main/resources/people.json'
|)""".stripMargin
sqlContext.sql(jsonDDL)
sqlContext.sql("describe jsonTable").collect()
sqlContext.sql("describe extended jsonTable").collect()
scala> sqlContext.sql("describe jsonTable").collect()
15/01/08 01:20:11 INFO sources.DDLParser: is extended ? -> false
res1: Array[org.apache.spark.sql.Row] = Array([# col_name,data_type,comment], [age,IntegerType,null], [name,StringType,null])
scala>
scala> sqlContext.sql("describe extended jsonTable").collect()
15/01/08 01:20:14 INFO sources.DDLParser: is extended ? -> true
res2: Array[org.apache.spark.sql.Row] = Array([# col_name,data_type,comment], [age,IntegerType,null], [name,StringType,null], [# extended,null,null])
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22762492
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/DDLTestSuit.scala ---
@@ -0,0 +1,74 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.sql.sources
+import org.apache.spark.sql._
+
+
+class DDLScanSource extends RelationProvider {
+ override def createRelation(
+ sqlContext: SQLContext,
+ parameters: Map[String, String]): BaseRelation = {
+ SimpleDDLScan(parameters("from").toInt, parameters("TO").toInt)(sqlContext)
+ }
+}
+
+case class SimpleDDLScan(from: Int, to: Int)(@transient val sqlContext: SQLContext)
+ extends TableScan {
+
+ override def schema =
+ StructType(Seq(StructField("name", StringType, nullable = false),
+ StructField("age", IntegerType, nullable = false)))
+
+ override def buildScan() = sqlContext.sparkContext.parallelize(from to to).
+ map(e => Row(s"people$e",e*2))
+}
+
+class DDLTestSuit extends DataSourceTest {
+ import caseInsensisitiveContext._
+
+ before {
+ sql(
+ """
+ |CREATE TEMPORARY TABLE ddlPeople
+ |USING org.apache.spark.sql.sources.DDLScanSource
+ |OPTIONS (
+ | From '1',
+ | To '10'
+ |)
+ """.stripMargin)
+ }
+
+ sqlTest(
+ "SELECT * FROM ddlPeople",
+ (1 to 10).map(e => Row(s"people$e",e*2)).toSeq)
+
+ sqlTest(
+ "describe ddlPeople",
+ Seq(
+ Row("name", "StringType", null),
--- End diff --
We should probably be using the `simpleString` instead of printing the type name here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70217508
[Test build #25639 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25639/consoleFull) for PR 3935 at commit [`521bbd7`](https://github.com/apache/spark/commit/521bbd73b83223534a1397aa936a6bab554da7d7).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69199795
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25229/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69455903
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25359/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by scwf <gi...@git.apache.org>.
Github user scwf commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70233294
@OopsOutOfMemory, you need revert unnecessary changes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69132527
[Test build #25198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25198/consoleFull) for PR 3935 at commit [`d0fe2d6`](https://github.com/apache/spark/commit/d0fe2d6404cbc27abca71ce9b42c5300577f4759).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69308519
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25312/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69484168
[Test build #25373 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25373/consoleFull) for PR 3935 at commit [`b25d32a`](https://github.com/apache/spark/commit/b25d32a93286c900f6bfc83b79fe7c8a81e1f08c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22656932
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala ---
@@ -77,6 +80,15 @@ private[sql] class DDLParser extends StandardTokenParsers with PackratParsers wi
CreateTableUsing(tableName, provider, opts)
}
+ /*
+ * describe [extended] table jsonTable
--- End diff --
Nit: `jsonTable` seems a little confusing, remove the comment or add some more detail description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on a diff in the pull request:
https://github.com/apache/spark/pull/3935#discussion_r22637899
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -326,4 +327,17 @@ private[sql] abstract class SparkStrategies extends QueryPlanner[SparkPlan] {
case _ => Nil
}
}
+
+ case class DDLStrategy(context: SQLContext) extends Strategy {
+ def apply(plan: LogicalPlan): Seq[SparkPlan] = plan match {
+ case describe: logical.DescribeCommand =>
+ val resolvedTable = context.executePlan(describe.table).analyzed
+ resolvedTable match {
+ case LogicalRelation(relation) =>
+ Seq(ExecutedCommand(DDLDescribeCommand(relation, describe.output, describe.isExtended)))
+ case o: logical.DescribeCommand =>
+ ExecutedCommand(DescribeCommand(planLater(o), describe.output)) :: Nil
--- End diff --
@scwf Thanks for the suggestion : )
I think in the implementation level, both are ok.
And In my point of view, the `RunnableCommand` is a command with a simple `action` that gernarlly `without side-effect`, like `add jar`, `add file`, `drop table`. You can refer `hive.execution.commands.scala`.
Why here I add a strategy is to handle the side-effect is Because the sql behavior of external datasource is not like common sql. Maybe it will cause side-effect. And use two level abstractions is more extendable I think.
You can look deep into `CommandStrategy`, the `cache command`, `explain command` is also two level abstraction.
What do you think? @chenghao-intel
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by OopsOutOfMemory <gi...@git.apache.org>.
Github user OopsOutOfMemory commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69197461
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-70217510
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25639/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69299939
[Test build #25301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25301/consoleFull) for PR 3935 at commit [`4a24144`](https://github.com/apache/spark/commit/4a24144d774f61c9d539dfb38baae369d3525aa0).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class DDLDescribeCommand(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5135][SQL] Add support for describe [ex...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3935#issuecomment-69303281
[Test build #25312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25312/consoleFull) for PR 3935 at commit [`8f25bd7`](https://github.com/apache/spark/commit/8f25bd7734c35349057c0b891517390b6ea902ad).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org