You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by xwu0226 <gi...@git.apache.org> on 2016/05/24 08:26:10 UTC
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX]remove 'list files' ...
GitHub user xwu0226 opened a pull request:
https://github.com/apache/spark/pull/13276
[SPARK-15431][SQL][HOTFIX]remove 'list files' testcase from CliSuite for now
## What changes were proposed in this pull request?
The test case `list files` added in `CliSuite` by PR #13212 can not run in some jenkins jobs after being merged.
However, some jenkins jobs can pass:
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.6/
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.4/
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.2/
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7/
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.3/
Others failed on this test case. Let me remove this test case for now.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xwu0226/spark SPARK-15431-clisuite
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13276.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13276
----
commit edeef1194d7b778bf60ae2047e169aa97e22c395
Author: Xin Wu <xi...@us.ibm.com>
Date: 2016-05-24T08:20:52Z
SPARK-15431: remove 'list files' testcase from CliSuite for now
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222005115
```
2016-05-24 13:31:31.0 - stdout> spark-sql> ADD FILE file:/home/jenkins/workspace/spark-master-test-maven-hadoop-
2016-05-24 13:31:31.004 - stdout> 2.7/sql/hive-thriftserver/target/scala-2.11/test-classes/data/files/small_kv.txt
2016-05-24 13:31:31.005 - stdout> > LIST FILES
2016-05-24 13:31:31.008 - stdout> > LIST FILE file:/home/jenkins/workspace/spark-master-test-maven-hadoop
2016-05-24 13:31:31.011 - stdout> -2.7/sql/hive-thriftserver/target/scala-2.11/test-classes/data/files/small_kv.tx
2016-05-24 13:31:31.011 - stdout> t
2016-05-24 13:31:31.103 - stdout> >
```
In the job-log, I found the whole command is too long and split into two lines. Thus, I just thinking that could be the reason why it always failed?
@cloud-fan @andrewor14 @xwu0226
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-221432018
I have tried mvn build and test with the build/mvn commands used in jenkins jobs for hadoop 2.2, ..., 2.7 and can not recreate the test case failure on my environment. I am wondering whether there is timing issue on the test servers. Since the failures are on different checkpoints on different jenkins job. For example, `LIST FILES` command output is not captured on [spark-branch-2.0-test-sbt-hadoop-2.6](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.0-test-sbt-hadoop-2.6/70/consoleFull), while `LIST FILE <filepath> `command output is not captured on [spark-master-test-maven-hadoop-2.7](https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-master-test-maven-hadoop-2.7/981/consoleFull).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX]remove 'list files' ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-221223341
**[Test build #3012 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3012/consoleFull)** for PR 13276 at commit [`edeef11`](https://github.com/apache/spark/commit/edeef1194d7b778bf60ae2047e169aa97e22c395).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222191581
@yhuai Thanks for merging the test case fix! I will open another PR to continue the test case. The problem is in the test code. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX]remove 'list files' ...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13276#discussion_r64446146
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala ---
@@ -238,23 +238,4 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
runCliWithin(2.minute, Seq("-e", "!echo \"This is a test for Spark-11624\";"))(
"" -> "This is a test for Spark-11624")
}
-
- test("list jars") {
--- End diff --
@andrewor14 Thanks! will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222193363
let me know when you have the PR. I will add you to the whitelist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222183379
It causes master builds to fail. I am merging it. Is the bug in the implementation or in our test code?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX]remove 'list files' ...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/13276#discussion_r64445218
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala ---
@@ -238,23 +238,4 @@ class CliSuite extends SparkFunSuite with BeforeAndAfterAll with Logging {
runCliWithin(2.minute, Seq("-e", "!echo \"This is a test for Spark-11624\";"))(
"" -> "This is a test for Spark-11624")
}
-
- test("list jars") {
--- End diff --
please use `ignore("list jars")` instead of removing them. There's a high chance that we'll forget to add them back after deleting them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX]remove 'list files' ...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-221202290
cc @liancheng @cloud-fan Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222183974
Merging to master and branch 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222192476
@yhuai By the way, is there a way the PR builder can trigger the test once I push the code, without wait for one of you to say OK to test. I think this will safe a lot of our time. and I can find issue earlier. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222009267
Yes. what Xiao found out seems reasonable because the jenkins jobs that have longer names fail while those with shorter names passed, such as `spark-master-test-sbt-*`.
Thanks, Xiao!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-221365150
I think it's not a hot bug as it doesn't break the PR build, if we can figure it out ASAP
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX]remove 'list files' ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-221200432
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX]remove 'list files' ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-221226649
**[Test build #3012 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3012/consoleFull)** for PR 13276 at commit [`edeef11`](https://github.com/apache/spark/commit/edeef1194d7b778bf60ae2047e169aa97e22c395).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-221680340
Adding to the above observation:
1. `LIST FILES` command output is not captured on all `spark-branch-2.0-test-*` jenkins jobs.
2. `LIST FILE file:/home/jenkins/workspace/spark-master-test-maven-hadoop-<version>/sql/hive-thriftserver/target/scala-2.11/test-classes/data/files/small_kv.txt` command output is not captured on all `spark-master-test-maven-*` jenkins jobs. This tells that the first command `LIST FILES` have passed the test.
3. The test cases passed on `spark-master-test-sbt-*` jenkins jobs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by xwu0226 <gi...@git.apache.org>.
Github user xwu0226 commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222033336
We found that I missed the `;` on the commands that are passed into the `runCliWithin(...)`. such that there is no output from the command.
But I am having an issue even after adding the `;` for each command. The verification for the expected phase, such as `small_kv.txt` is already in the `ADD FILE/JAR` command, so the verification may still think it is successful even though the LIST command may not return the expected results.
I have done the manual spark-sql console test for these commands. But I have not been able to find a way to do the verification automatically according to the above reason. Any suggestions!! Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13276
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/13276#issuecomment-222185524
@xwu0226 Please open another PR to re-enable these tests and ask jenkins PR builder to test maven.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org