How/where to set the hostname used by the spark workers?

[Gerard Maas <ge...@gmail.com>]

Hi,

I'm trying to run Spark in Docker, using the amplab docker scripts (which
I've been modifying  to support 0.9.0)

I'm trying to use Docker's own link facility instead of the provided DNS
service to have master-worker communication, using plain IP addresses.

Right now, the master is working fine, but the workers are picking up the
hostname when they build the remote actor address:

INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkWorker@*devworker*:54621]

Where 'devworker' is the name given to  the docker container and
non-routable from other containers.

For the master, by setting  `SPARK_MASTER_IP` on the `spark_env.sh`, it's
working fine:
INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkMaster@*172.17.0.41*:7077]

Yet, there's no SPARK_WORKER_IP option there.

How could I instruct the spark worker to use a given ip address in a
similar fashion?

Thanks,

Gerard.

Re: How/where to set the hostname used by the spark workers?

[Gerard Maas <ge...@gmail.com>]

Hi,

Thanks.  For the record, I found a solution by passing the ip address as
parameter:

spark-class org.apache.spark.deploy.worker.Worker $MASTER -i $1

-kr, Gerard.



On Tue, Feb 11, 2014 at 7:20 PM, Soumya Simanta <so...@gmail.com>wrote:

> try setting worker/slaves ip on $SPARK_HOME/conf/slaves
>
>
> On Tue, Feb 11, 2014 at 1:14 PM, Gerard Maas <ge...@gmail.com>wrote:
>
>> Hi,
>>
>> I'm trying to run Spark in Docker, using the amplab docker scripts (which
>> I've been modifying  to support 0.9.0)
>>
>> I'm trying to use Docker's own link facility instead of the provided DNS
>> service to have master-worker communication, using plain IP addresses.
>>
>> Right now, the master is working fine, but the workers are picking up the
>> hostname when they build the remote actor address:
>>
>> INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkWorker@*devworker*:54621]
>>
>> Where 'devworker' is the name given to  the docker container and
>> non-routable from other containers.
>>
>> For the master, by setting  `SPARK_MASTER_IP` on the `spark_env.sh`, it's
>> working fine:
>> INFO Remoting: Remoting started; listening on addresses
>> :[akka.tcp://sparkMaster@*172.17.0.41*:7077]
>>
>> Yet, there's no SPARK_WORKER_IP option there.
>>
>> How could I instruct the spark worker to use a given ip address in a
>> similar fashion?
>>
>> Thanks,
>>
>> Gerard.
>>
>
>

Re: How/where to set the hostname used by the spark workers?

[Soumya Simanta <so...@gmail.com>]

try setting worker/slaves ip on $SPARK_HOME/conf/slaves


On Tue, Feb 11, 2014 at 1:14 PM, Gerard Maas <ge...@gmail.com> wrote:

> Hi,
>
> I'm trying to run Spark in Docker, using the amplab docker scripts (which
> I've been modifying  to support 0.9.0)
>
> I'm trying to use Docker's own link facility instead of the provided DNS
> service to have master-worker communication, using plain IP addresses.
>
> Right now, the master is working fine, but the workers are picking up the
> hostname when they build the remote actor address:
>
> INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkWorker@*devworker*:54621]
>
> Where 'devworker' is the name given to  the docker container and
> non-routable from other containers.
>
> For the master, by setting  `SPARK_MASTER_IP` on the `spark_env.sh`, it's
> working fine:
> INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkMaster@*172.17.0.41*:7077]
>
> Yet, there's no SPARK_WORKER_IP option there.
>
> How could I instruct the spark worker to use a given ip address in a
> similar fashion?
>
> Thanks,
>
> Gerard.
>