You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liancheng <gi...@git.apache.org> on 2015/02/25 05:35:36 UTC
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/4758
[SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PID_DIR when running Thrift server test suites
This is a follow-up of #4720. By default, `spark-daemon.sh` writes PID files under `/tmp`, which makes it impossible to start multiple server instances simultaneously. This PR sets `SPARK_PID_DIR` to Spark home directory to workaround this problem.
Many thanks to @chenghao-intel for pointing out this issue!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark thriftserver-pid-dir
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4758.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4758
----
commit 1b3d1e3f7215c09aefe28bc0c0b75c3e09de3f8d
Author: Cheng Lian <li...@databricks.com>
Date: 2015-02-25T04:31:58Z
Sets SPARK_HOME as SPARK_PID_DIR when running Thrift server test suites
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76499219
@andrewor14 I just merged it to master and branch-1.3.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76284089
I'm going to merge this with @chenghao-intel's suggestion. Thanks everyone. This is going into master 1.3
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76444880
[Test build #28077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28077/consoleFull) for PR 4758 at commit [`252fa0f`](https://github.com/apache/spark/commit/252fa0f70a93be81cf7b1151d720b5b7703cc011).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25318071
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode, attempt=$attempt")
- logPath = Process(command, None, "SPARK_TESTING" -> "0").lines.collectFirst {
+ val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
I am wondering a better path would be `Utils.createTempDir(..)`, just in case we want to run the test suite simultaneously in the local machine as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25485273
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode, attempt=$attempt")
- logPath = Process(command, None, "SPARK_TESTING" -> "0").lines.collectFirst {
+ val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
oh you're right. Looks like we currently set it to `/tmp` already, which is probably what `java.io.tmpdir` is anyway in most cases
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4758
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25469585
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode, attempt=$attempt")
- logPath = Process(command, None, "SPARK_TESTING" -> "0").lines.collectFirst {
+ val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
actually, can we just use `sys.props(java.io.tmpdir)` instead of creating a new directory?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25526262
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode, attempt=$attempt")
- logPath = Process(command, None, "SPARK_TESTING" -> "0").lines.collectFirst {
+ val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
Updated, you can merge it after Jenkins nods :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76503823
Cool, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-75909373
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27935/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76456868
[Test build #28077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28077/consoleFull) for PR 4758 at commit [`252fa0f`](https://github.com/apache/spark/commit/252fa0f70a93be81cf7b1151d720b5b7703cc011).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by chenghao-intel <gi...@git.apache.org>.
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25479512
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode, attempt=$attempt")
- logPath = Process(command, None, "SPARK_TESTING" -> "0").lines.collectFirst {
+ val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
We need a different directory for each run, otherwise the PID file will be overwritten when multiple jenkins unit tests run parallel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76456876
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28077/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-75904355
[Test build #27935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27935/consoleFull) for PR 4758 at commit [`1b3d1e3`](https://github.com/apache/spark/commit/1b3d1e3f7215c09aefe28bc0c0b75c3e09de3f8d).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-75909369
[Test build #27935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27935/consoleFull) for PR 4758 at commit [`1b3d1e3`](https://github.com/apache/spark/commit/1b3d1e3f7215c09aefe28bc0c0b75c3e09de3f8d).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76284574
I said I was going to merge this, but instead I left one more comment inline.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/4758#issuecomment-76444830
Updated, you can merge it after Jenkins nods :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25414445
--- Diff: sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala ---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode, attempt=$attempt")
- logPath = Process(command, None, "SPARK_TESTING" -> "0").lines.collectFirst {
+ val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
Good point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org