You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Fernando O." <fo...@gmail.com> on 2014/12/30 19:23:52 UTC
Trying to make spark-jobserver work with yarn
Hi all,
I'm investigating spark for a new project and I'm trying to use
spark-jobserver because... I need to reuse and share RDDs and from what I
read in the forum that's the "standard" :D
Turns out that spark-jobserver doesn't seem to work on yarn, or at least it
does not on 1.1.1
My config is spark 1.1.1 (moving to 1.2.0 soon), hadoop 2.6 (which seems
compatible with 2.4 from spark point of view... at least I was able to run
spark-submit and shell tasks both in yarn-client and yarn-cluster modes)
going back to my original point, I did some changes in spark-jobserver and
how I can submit a job but I get:
....
[2014-12-30 18:20:19,769] INFO e.spark.deploy.yarn.Client []
[akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
- Max mem capabililty of a single resource in this cluster 15000
[2014-12-30 18:20:19,770] INFO e.spark.deploy.yarn.Client []
[akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
- Preparing Local resources
[2014-12-30 18:20:20,041] INFO e.spark.deploy.yarn.Client []
[akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
- Prepared Local resources Map(__spark__.jar -> resource { scheme: "file"
port: -1 file:
"/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar"
} size: 343226 timestamp: 1416429031000 type: FILE visibility: PRIVATE)
[...]
[2014-12-30 18:20:20,139] INFO e.spark.deploy.yarn.Client []
[akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
- Yarn AM launch context:
[2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
[akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
- class: org.apache.spark.deploy.yarn.ExecutorLauncher
[2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
[akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
- env: Map(CLASSPATH ->
$PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:$PWD/__app__.jar:$PWD/*,
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 343226, SPARK_YARN_STAGING_DIR ->
.sparkStaging/application_1419963137232_0001/,
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_USER -> ec2-user,
SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS ->
1416429031000, SPARK_YARN_CACHE_FILES ->
file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar#__spark__.jar)
[...]
[2014-12-30 18:03:04,474] INFO YarnClientSchedulerBackend []
[akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
- Application report from ASM:
appMasterRpcPort: -1
appStartTime: 1419962580444
yarnAppState: FAILED
[2014-12-30 18:03:04,475] ERROR .jobserver.JobManagerActor []
[akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
- Failed to create context ebac0153-spark.jobserver.WordCountExample,
shutting down actor
org.apache.spark.SparkException: Yarn application already ended,might be
killed or not able to launch application master.
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApp(YarnClientSchedulerBackend.scala:117)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:93)
In the hadoop console I can get the detailed issue
Diagnostics: File
file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
does not exist
java.io.FileNotFoundException: File
file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
does not exist
now... it seems like spark is actually use a file I used for launching the
task in other nodes
Can anyone point me in the right direction of where that might be being set?
Re: Trying to make spark-jobserver work with yarn
Posted by "Fernando O." <fo...@gmail.com>.
Thanks Akhil, that will help a lot !
It turned out that spark-jobserver does not work in "development mode" but
if you deploy a server it works (looks like the dependencies when running
jobserver from sbt are not right)
On Thu, Jan 1, 2015 at 5:22 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Hi Fernando,
>
> Here's a <https://github.com/sigmoidanalytics/Test> simple log
> parser/analyser written in scala (you can run it without
> spark-shell/submit). https://github.com/sigmoidanalytics/Test
>
> Basically to run a spark job without spark-submit or shell you need a build
> file <https://github.com/sigmoidanalytics/Test/blob/master/build.sbt>
> which will pull in all the dependecies, and the main program
> <https://github.com/sigmoidanalytics/Test/blob/master/src/main/scala/LogAnalyser.scala#L35>
> in which you will specify your cluster details while creating the
> SparkContext.
>
> Thanks
> Best Regards
>
> On Wed, Dec 31, 2014 at 10:54 PM, Fernando O. <fo...@gmail.com> wrote:
>
>> Before jumping into a sea of dependencies and bash files:
>> Does anyone have an example of how to run a spark job without using
>> spark-submit or shell ?
>>
>> On Tue, Dec 30, 2014 at 3:23 PM, Fernando O. <fo...@gmail.com> wrote:
>>
>>> Hi all,
>>> I'm investigating spark for a new project and I'm trying to use
>>> spark-jobserver because... I need to reuse and share RDDs and from what I
>>> read in the forum that's the "standard" :D
>>>
>>> Turns out that spark-jobserver doesn't seem to work on yarn, or at least
>>> it does not on 1.1.1
>>>
>>> My config is spark 1.1.1 (moving to 1.2.0 soon), hadoop 2.6 (which seems
>>> compatible with 2.4 from spark point of view... at least I was able to run
>>> spark-submit and shell tasks both in yarn-client and yarn-cluster modes)
>>>
>>>
>>>
>>>
>>> going back to my original point, I did some changes in spark-jobserver
>>> and how I can submit a job but I get:
>>>
>>> ....
>>> [2014-12-30 18:20:19,769] INFO e.spark.deploy.yarn.Client []
>>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>>> - Max mem capabililty of a single resource in this cluster 15000
>>> [2014-12-30 18:20:19,770] INFO e.spark.deploy.yarn.Client []
>>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>>> - Preparing Local resources
>>> [2014-12-30 18:20:20,041] INFO e.spark.deploy.yarn.Client []
>>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>>> - Prepared Local resources Map(__spark__.jar -> resource { scheme: "file"
>>> port: -1 file:
>>> "/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar"
>>> } size: 343226 timestamp: 1416429031000 type: FILE visibility: PRIVATE)
>>>
>>> [...]
>>>
>>> [2014-12-30 18:20:20,139] INFO e.spark.deploy.yarn.Client []
>>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>>> - Yarn AM launch context:
>>> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
>>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>>> - class: org.apache.spark.deploy.yarn.ExecutorLauncher
>>> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
>>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>>> - env: Map(CLASSPATH ->
>>> $PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:$PWD/__app__.jar:$PWD/*,
>>> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 343226, SPARK_YARN_STAGING_DIR ->
>>> .sparkStaging/application_1419963137232_0001/,
>>> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_USER -> ec2-user,
>>> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS ->
>>> 1416429031000, SPARK_YARN_CACHE_FILES ->
>>> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar#__spark__.jar)
>>>
>>> [...]
>>>
>>> [2014-12-30 18:03:04,474] INFO YarnClientSchedulerBackend []
>>> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
>>> - Application report from ASM:
>>> appMasterRpcPort: -1
>>> appStartTime: 1419962580444
>>> yarnAppState: FAILED
>>>
>>> [2014-12-30 18:03:04,475] ERROR .jobserver.JobManagerActor []
>>> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
>>> - Failed to create context ebac0153-spark.jobserver.WordCountExample,
>>> shutting down actor
>>> org.apache.spark.SparkException: Yarn application already ended,might be
>>> killed or not able to launch application master.
>>> at
>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApp(YarnClientSchedulerBackend.scala:117)
>>> at
>>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:93)
>>>
>>>
>>>
>>> In the hadoop console I can get the detailed issue
>>>
>>> Diagnostics: File
>>> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
>>> does not exist
>>> java.io.FileNotFoundException: File
>>> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
>>> does not exist
>>>
>>> now... it seems like spark is actually use a file I used for launching
>>> the task in other nodes
>>>
>>> Can anyone point me in the right direction of where that might be being
>>> set?
>>>
>>>
>>
>
Re: Trying to make spark-jobserver work with yarn
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Hi Fernando,
Here's a <https://github.com/sigmoidanalytics/Test> simple log
parser/analyser written in scala (you can run it without
spark-shell/submit). https://github.com/sigmoidanalytics/Test
Basically to run a spark job without spark-submit or shell you need a build
file <https://github.com/sigmoidanalytics/Test/blob/master/build.sbt> which
will pull in all the dependecies, and the main program
<https://github.com/sigmoidanalytics/Test/blob/master/src/main/scala/LogAnalyser.scala#L35>
in which you will specify your cluster details while creating the
SparkContext.
Thanks
Best Regards
On Wed, Dec 31, 2014 at 10:54 PM, Fernando O. <fo...@gmail.com> wrote:
> Before jumping into a sea of dependencies and bash files:
> Does anyone have an example of how to run a spark job without using
> spark-submit or shell ?
>
> On Tue, Dec 30, 2014 at 3:23 PM, Fernando O. <fo...@gmail.com> wrote:
>
>> Hi all,
>> I'm investigating spark for a new project and I'm trying to use
>> spark-jobserver because... I need to reuse and share RDDs and from what I
>> read in the forum that's the "standard" :D
>>
>> Turns out that spark-jobserver doesn't seem to work on yarn, or at least
>> it does not on 1.1.1
>>
>> My config is spark 1.1.1 (moving to 1.2.0 soon), hadoop 2.6 (which seems
>> compatible with 2.4 from spark point of view... at least I was able to run
>> spark-submit and shell tasks both in yarn-client and yarn-cluster modes)
>>
>>
>>
>>
>> going back to my original point, I did some changes in spark-jobserver
>> and how I can submit a job but I get:
>>
>> ....
>> [2014-12-30 18:20:19,769] INFO e.spark.deploy.yarn.Client []
>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>> - Max mem capabililty of a single resource in this cluster 15000
>> [2014-12-30 18:20:19,770] INFO e.spark.deploy.yarn.Client []
>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>> - Preparing Local resources
>> [2014-12-30 18:20:20,041] INFO e.spark.deploy.yarn.Client []
>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>> - Prepared Local resources Map(__spark__.jar -> resource { scheme: "file"
>> port: -1 file:
>> "/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar"
>> } size: 343226 timestamp: 1416429031000 type: FILE visibility: PRIVATE)
>>
>> [...]
>>
>> [2014-12-30 18:20:20,139] INFO e.spark.deploy.yarn.Client []
>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>> - Yarn AM launch context:
>> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>> - class: org.apache.spark.deploy.yarn.ExecutorLauncher
>> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
>> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
>> - env: Map(CLASSPATH ->
>> $PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:$PWD/__app__.jar:$PWD/*,
>> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 343226, SPARK_YARN_STAGING_DIR ->
>> .sparkStaging/application_1419963137232_0001/,
>> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_USER -> ec2-user,
>> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS ->
>> 1416429031000, SPARK_YARN_CACHE_FILES ->
>> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar#__spark__.jar)
>>
>> [...]
>>
>> [2014-12-30 18:03:04,474] INFO YarnClientSchedulerBackend []
>> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
>> - Application report from ASM:
>> appMasterRpcPort: -1
>> appStartTime: 1419962580444
>> yarnAppState: FAILED
>>
>> [2014-12-30 18:03:04,475] ERROR .jobserver.JobManagerActor []
>> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
>> - Failed to create context ebac0153-spark.jobserver.WordCountExample,
>> shutting down actor
>> org.apache.spark.SparkException: Yarn application already ended,might be
>> killed or not able to launch application master.
>> at
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApp(YarnClientSchedulerBackend.scala:117)
>> at
>> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:93)
>>
>>
>>
>> In the hadoop console I can get the detailed issue
>>
>> Diagnostics: File
>> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
>> does not exist
>> java.io.FileNotFoundException: File
>> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
>> does not exist
>>
>> now... it seems like spark is actually use a file I used for launching
>> the task in other nodes
>>
>> Can anyone point me in the right direction of where that might be being
>> set?
>>
>>
>
Re: Trying to make spark-jobserver work with yarn
Posted by "Fernando O." <fo...@gmail.com>.
Before jumping into a sea of dependencies and bash files:
Does anyone have an example of how to run a spark job without using
spark-submit or shell ?
On Tue, Dec 30, 2014 at 3:23 PM, Fernando O. <fo...@gmail.com> wrote:
> Hi all,
> I'm investigating spark for a new project and I'm trying to use
> spark-jobserver because... I need to reuse and share RDDs and from what I
> read in the forum that's the "standard" :D
>
> Turns out that spark-jobserver doesn't seem to work on yarn, or at least
> it does not on 1.1.1
>
> My config is spark 1.1.1 (moving to 1.2.0 soon), hadoop 2.6 (which seems
> compatible with 2.4 from spark point of view... at least I was able to run
> spark-submit and shell tasks both in yarn-client and yarn-cluster modes)
>
>
>
>
> going back to my original point, I did some changes in spark-jobserver and
> how I can submit a job but I get:
>
> ....
> [2014-12-30 18:20:19,769] INFO e.spark.deploy.yarn.Client []
> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
> - Max mem capabililty of a single resource in this cluster 15000
> [2014-12-30 18:20:19,770] INFO e.spark.deploy.yarn.Client []
> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
> - Preparing Local resources
> [2014-12-30 18:20:20,041] INFO e.spark.deploy.yarn.Client []
> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
> - Prepared Local resources Map(__spark__.jar -> resource { scheme: "file"
> port: -1 file:
> "/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar"
> } size: 343226 timestamp: 1416429031000 type: FILE visibility: PRIVATE)
>
> [...]
>
> [2014-12-30 18:20:20,139] INFO e.spark.deploy.yarn.Client []
> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
> - Yarn AM launch context:
> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
> - class: org.apache.spark.deploy.yarn.ExecutorLauncher
> [2014-12-30 18:20:20,140] INFO e.spark.deploy.yarn.Client []
> [akka://JobServer/user/context-supervisor/f983d86e-spark.jobserver.WordCountExample]
> - env: Map(CLASSPATH ->
> $PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:$PWD/__app__.jar:$PWD/*,
> SPARK_YARN_CACHE_FILES_FILE_SIZES -> 343226, SPARK_YARN_STAGING_DIR ->
> .sparkStaging/application_1419963137232_0001/,
> SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE, SPARK_USER -> ec2-user,
> SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS ->
> 1416429031000, SPARK_YARN_CACHE_FILES ->
> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar#__spark__.jar)
>
> [...]
>
> [2014-12-30 18:03:04,474] INFO YarnClientSchedulerBackend []
> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
> - Application report from ASM:
> appMasterRpcPort: -1
> appStartTime: 1419962580444
> yarnAppState: FAILED
>
> [2014-12-30 18:03:04,475] ERROR .jobserver.JobManagerActor []
> [akka://JobServer/user/context-supervisor/ebac0153-spark.jobserver.WordCountExample]
> - Failed to create context ebac0153-spark.jobserver.WordCountExample,
> shutting down actor
> org.apache.spark.SparkException: Yarn application already ended,might be
> killed or not able to launch application master.
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApp(YarnClientSchedulerBackend.scala:117)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:93)
>
>
>
> In the hadoop console I can get the detailed issue
>
> Diagnostics: File
> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
> does not exist
> java.io.FileNotFoundException: File
> file:/home/ec2-user/.ivy2/cache/org.apache.spark/spark-yarn_2.10/jars/spark-yarn_2.10-1.1.1.jar
> does not exist
>
> now... it seems like spark is actually use a file I used for launching the
> task in other nodes
>
> Can anyone point me in the right direction of where that might be being
> set?
>
>