You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tianyi <gi...@git.apache.org> on 2014/08/04 07:15:38 UTC
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
GitHub user tianyi opened a pull request:
https://github.com/apache/spark/pull/1760
[SPARK-2817] add "show create table" support
In spark sql component, the "show create table" syntax had been disabled.
We thought it is a useful funciton to describe a hive table.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tianyi/spark spark-2817
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1760.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1760
----
commit a03db77bad7aa4deb82ce23f9958e43394e9752b
Author: tianyi <ti...@asiainfo-linkage.com>
Date: 2014-08-04T05:13:01Z
[SPARK-2817] add "show create table" support
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51018444
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-52127169
QA results for PR 1760:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18487/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51140547
Hi, Michael. I run the test on my laptop with this configuration, it pass the test which failed before. I also check that the apache-hive project create the same configuration in pom.xml.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51369980
Yeah, but then we have to handle escaping and such ourselves and if there are other properties the function could get unwieldily. Is there a problem calling `System.setProperty(...)` in the constructor of `TestHive`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51119298
Can you please add [SQL] to the title of any PRs that affect Spark SQL?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15979361
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -95,7 +101,9 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
* Replaces relative paths to the parent directory "../" with hiveDevHome since this is how the
* hive test cases assume the system is set up.
*/
- private def rewritePaths(cmd: String): String =
+ private def rewritePaths(cmd: String): String = {
+ // For some hive test case which contain ${system:test.tmp.dir}
+ System.setProperty("test.tmp.dir", testTmpDir.getCanonicalPath)
--- End diff --
it define on line:73
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-52117673
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51995629
@marmbrus I think the test failed because some enviorment problem on jenkins host. could you run it again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51318897
how about adding another rule in rewritePaths funciton in TestHive.scala?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51996793
QA tests have started for PR 1760. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18399/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r16066399
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -70,6 +72,17 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
set("hive.metastore.warehouse.dir", warehousePath)
}
+ val testTmpDir = if (System.getProperty("user.dir").endsWith("sql" +
--- End diff --
Could this just be:
```scala
val testTempDir = File.createtempfile("", "spark.hive.tmp")
testTempDir.delete()
testTempDir.mkdir()
```
I'm not sure why we would want to keep temporary files in the source tree instead of the system temporary folder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51018501
Don't forget to commit the golden answer files.
You can get that by running "sbt/sbt -Phive=true hive/test".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51869591
QA tests have started for PR 1760. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18355/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1760
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51567841
@chenghao-intel I did not understand what‘s your mean about "add the file"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51075309
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15979304
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -95,7 +101,9 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
* Replaces relative paths to the parent directory "../" with hiveDevHome since this is how the
* hive test cases assume the system is set up.
*/
- private def rewritePaths(cmd: String): String =
+ private def rewritePaths(cmd: String): String = {
+ // For some hive test case which contain ${system:test.tmp.dir}
+ System.setProperty("test.tmp.dir", testTmpDir.getCanonicalPath)
--- End diff --
I didn't see anywhere use the `test.tmp.dir`, am I missed something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51072215
what's wrong with jenkins?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15979199
--- Diff: sql/hive/pom.xml ---
@@ -139,6 +139,11 @@
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
--- End diff --
Shall we remove this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51567277
@chenghao-intel , I had added a line “last_modified_by” in nonDeterministicLineIndicators
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51119263
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51139532
I modified the pom.xml in spark-hive module, to support ${system:test.tmp.dir}.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51996403
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-52121285
QA tests have started for PR 1760. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18487/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51876493
Be aware that the `udf_unix_timestamp` case is timezone sensitive. That's why we reset timezone to "America/Los_Angeles" in `beforeAll`. This may be related to your test failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15979301
--- Diff: sql/hive/pom.xml ---
@@ -139,6 +139,11 @@
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
+ <configuration>
--- End diff --
sorry, I forgot to remove this line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15979418
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -95,7 +101,9 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
* Replaces relative paths to the parent directory "../" with hiveDevHome since this is how the
* hive test cases assume the system is set up.
*/
- private def rewritePaths(cmd: String): String =
+ private def rewritePaths(cmd: String): String = {
+ // For some hive test case which contain ${system:test.tmp.dir}
+ System.setProperty("test.tmp.dir", testTmpDir.getCanonicalPath)
--- End diff --
sorry , the “test.tmp.dir” used in some hive test sql， like：show_create_table_delimited.q
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15981109
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -95,7 +101,9 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
* Replaces relative paths to the parent directory "../" with hiveDevHome since this is how the
* hive test cases assume the system is set up.
*/
- private def rewritePaths(cmd: String): String =
+ private def rewritePaths(cmd: String): String = {
+ // For some hive test case which contain ${system:test.tmp.dir}
+ System.setProperty("test.tmp.dir", testTmpDir.getCanonicalPath)
--- End diff --
OK, make sense.
* Maybe it's better move this line out of function rewritePaths, as it may be called multiple times.
* In addition, we also have to delete the temporal folder after unit test finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51083724
LGTM :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r16066159
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -60,6 +60,8 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
// without restarting the JVM.
System.clearProperty("spark.hostPort")
+
+
--- End diff --
Can you remove these spurious new line additions? Below as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15981282
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -70,6 +70,12 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
set("hive.metastore.warehouse.dir", warehousePath)
}
+ val testTmpDir = if (System.getProperty("user.dir").endsWith("sql" + File.separator + "hive")) {
+ new File(System.getProperty("user.dir") + File.separator + "tmp")
--- End diff --
All we need is the temporal folder, not sure why we have to check the user working directory if end with `sql/hive`,
Can we just use `val testTmpDir = new File(System.getProperty("user.dir") + File.separator + "tmp")` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51137779
there are two problem:
1 the result of "show create table" contains some time properties, which could not be the same with the time of the case running. Do we have hadoop and hive environment on jenkins machine? Should I remove some golden files?
2 I could not find how to get the ${system:test.tmp.dir} variable on my laptop, I thought it could be a environment related problem, but it failed on jenkins too. I think I can fix this by modify the pom.xml in "sql/hive/".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51419063
2 question:
Q1: There are few variable in hive test sql, like:
system:build.ivy.lib.dir
system:test.dfs.mkdir
system:test.src.data.dir
system:test.tmp.dir
system:xxx
should I add them all in TestHive ?
Q2: In Hive project, the "test.tmp.dir" had been set to "${project.build.directory}/tmp" , I check all the SystemProperties , the "user.dir" is the closest one to ${project.build.directory}. Should I find best value for other variable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51428164
I just add "${system:test.tmp.dir}" support for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-52000858
QA results for PR 1760:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18399/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51021277
@chenghao-intel is these files all right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51875106
QA tests have started for PR 1760. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18359/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15981816
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -95,7 +101,9 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
* Replaces relative paths to the parent directory "../" with hiveDevHome since this is how the
* hive test cases assume the system is set up.
*/
- private def rewritePaths(cmd: String): String =
+ private def rewritePaths(cmd: String): String = {
+ // For some hive test case which contain ${system:test.tmp.dir}
+ System.setProperty("test.tmp.dir", testTmpDir.getCanonicalPath)
--- End diff --
@chenghao-intel
1 how about move it after define the testTmpDir?
2 may I add some code in reset function to delete the temporal folder after unit test finished?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-52127347
This only failed streaming tests. I'm going to merge into master and 1.1.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51567887
Sorry, I mean add the field `file`, see `'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51140765
Another option would be to set this system property somewhere in the HiveComparisionTest instead of in the build file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51869764
QA results for PR 1760:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18355/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51567155
@tianyi sorry, I couldn't open the last failure log of the unit test, if it failed in the dynamic value comparison, you can add the property in the [nonDeterministicLineIndicators](https://github.com/apache/spark/blob/master/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala#L159) as @marmbrus said.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51567507
I saw that, in order not to modify the `TestHive.scala`, properly we can also add the `file`.
PS: From the golden file
```
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'file:/tmp/sparkHiveWarehouse1280221975983654134/tmp_showcrt1'
TBLPROPERTIES (
'EXTERNAL'='FALSE',
'last_modified_by'='tianyi',
'last_modified_time'='1407132100',
'transient_lastDdlTime'='1407132100')
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-52120876
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51569492
@chenghao-intel there is a function "cleanPaths" to clean these for ignore filePath difference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r16005429
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -70,6 +70,12 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
set("hive.metastore.warehouse.dir", warehousePath)
}
+ val testTmpDir = if (System.getProperty("user.dir").endsWith("sql" + File.separator + "hive")) {
+ new File(System.getProperty("user.dir") + File.separator + "tmp")
--- End diff --
in repo tests is checking to see if the tests are there or if we should look for them in the hive distribution (they weren't always bundled with the source tree). If we need a temporary directory, we should use the mechanisms provided by the JVM for that (`File.createTempFile`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15981137
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -70,6 +70,12 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
set("hive.metastore.warehouse.dir", warehousePath)
}
+ val testTmpDir = if (System.getProperty("user.dir").endsWith("sql" + File.separator + "hive")) {
+ new File(System.getProperty("user.dir") + File.separator + "tmp")
+ } else {
+ new File(System.getProperty("user.dir") + File.separator + "sql" + File.separator + "hive" + File.separator + "tmp")
--- End diff --
The line length greater than 100? (run `sbt/sbt -Phive scalastyle`)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-52010613
@marmbrus Could you ask someone help me to fix this error ?
Failed example:
srdd.collect()
Exception raised:
Traceback (most recent call last):
File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
compileflags, 1) in test.globs
File "<doctest pyspark.sql.SQLContext.inferSchema[6]>", line 1, in <module>
srdd.collect()
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1613, in collect
rows = RDD.collect(self)
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/rdd.py", line 724, in collect
bytesInJava = self._jrdd.collect().iterator()
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
self.target_id, self.name)
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
format(target_id, '.', name), value)
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 35.0 failed 1 times, most recent failure: Lost task 1.0 in stage 35.0 (TID 72, localhost): java.lang.ClassCastException: java.lang.String cannot be cast to java.util.ArrayList
net.razorvine.pickle.objects.ArrayConstructor.construct(ArrayConstructor.java:33)
net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:617)
net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:170)
net.razorvine.pickle.Unpickler.load(Unpickler.java:84)
net.razorvine.pickle.Unpickler.loads(Unpickler.java:97)
......
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15981486
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -70,6 +70,12 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
set("hive.metastore.warehouse.dir", warehousePath)
}
+ val testTmpDir = if (System.getProperty("user.dir").endsWith("sql" + File.separator + "hive")) {
+ new File(System.getProperty("user.dir") + File.separator + "tmp")
--- End diff --
@chenghao-intel I did not understand why we have to check that too. I just follow the same way of handling "inRepoTests"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51869442
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51140708
Oh, I see, its a system property that is set by maven and then threaded though to hive. Thank you for explaining.
I'd still just skip this test as thats not going to work for sbt (which is what Jenkins uses), and I don't want to have to change all the build files for this small detail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51119762
QA tests have started for PR 1760. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17876/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51025307
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/1760#discussion_r15985611
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala ---
@@ -95,7 +101,9 @@ class TestHiveContext(sc: SparkContext) extends HiveContext(sc) {
* Replaces relative paths to the parent directory "../" with hiveDevHome since this is how the
* hive test cases assume the system is set up.
*/
- private def rewritePaths(cmd: String): String =
+ private def rewritePaths(cmd: String): String = {
+ // For some hive test case which contain ${system:test.tmp.dir}
+ System.setProperty("test.tmp.dir", testTmpDir.getCanonicalPath)
--- End diff --
@chenghao-intel
I think we do not have to delete the temporal folder because we already delete created table after unit test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51138102
We can add time / user specific properties to [nonDeterministicLineIndicators](https://github.com/apache/spark/blob/master/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala#L159)
Regarding the test that requires a system property, we don't support that yet. I'd remove that test from the whitelist.
Another note: there is no hive/hadoop on jenkins so only tests with golden files will pass there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51565490
Hi Michael , could you review these codes again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51139602
I don't think thats a maven thing is it? I think thats a hive substitution that we haven't implemented.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51878493
QA results for PR 1760:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18359/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51739210
@chenghao-intel @marmbrus If there is any problem about this PR, please let me know. thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] [SQL] add "show create table" sup...
Posted by tianyi <gi...@git.apache.org>.
Github user tianyi commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51875134
I'm sorry for forgetting run the test yesterday.
this time, i passed all the test on my laptop except udf_unix_timestamp function, I guess it should be a enviorment problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2817] add "show create table" support
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1760#issuecomment-51128549
QA results for PR 1760:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds no public classes<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17876/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org