You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Aaron <aa...@gmail.com> on 2015/12/16 14:00:55 UTC
Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for
client mode
In going through running various Spark jobs, both Spark 1.5.2 and the
new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
noticed that is in order to run the Spark shells (both python and
scala), we needed to set the LIBPROCESS_IP environment variable before
running.
Was curious if the Spark on Mesos docs should be updated, under the
Client Mode section, to include setting this environment variable?
Cheers
Aaron
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Iulian Dragoș <iu...@typesafe.com>.
On Wed, Dec 16, 2015 at 5:42 PM, Aaron <aa...@gmail.com> wrote:
> Wrt to PR, sure, let me update the documentation, i'll send it out
> shortly. My Fork is on Github..is the PR from there ok?
>
Absolutely. Have a look at
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark if
you haven't done so already, it should answer most questions about starting
to contribute to Spark.
thanks,
iulian
>
> Cheers,
> Aaron
>
> On Wed, Dec 16, 2015 at 11:33 AM, Timothy Chen <tn...@gmail.com> wrote:
> > Yes if want to manually override what IP to use to be contacted by the
> > master you can set LIPROCESS_IP and LIBPROCESS_PORT.
> >
> > It is a Mesos specific settings. We can definitely update the docs.
> >
> > Note that in the future as we move to use the new Mesos Http API these
> > configurations won't be needed (also libmesos!).
> >
> > Tim
> >
> > On Dec 16, 2015, at 8:09 AM, Iulian Dragoș <iu...@typesafe.com>
> > wrote:
> >
> > LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a
> > Mesos-specific setting.
> >
> > Have you tried setting SPARK_LOCAL_IP?
> >
> > On Wed, Dec 16, 2015 at 5:07 PM, Aaron <aa...@gmail.com> wrote:
> >>
> >> Found this thread that talked about it to help understand it better:
> >>
> >>
> >>
> https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
> >>
> >> >
> >> > When you run Spark on Mesos it needs to run
> >> >
> >> > spark driver
> >> > mesos scheduler
> >> >
> >> > and both need to be visible to outside world on public iface IP
> >> >
> >> > you need to tell Spark and Mesos on which interface to bind - by
> default
> >> > they resolve node hostname to ip - this is loopback address in your
> case
> >> >
> >> > Possible solutions - on slave node with public IP 192.168.56.50
> >> >
> >> > 1. Set
> >> >
> >> > export LIBPROCESS_IP=192.168.56.50
> >> > export SPARK_LOCAL_IP=192.168.56.50
> >> >
> >> > 2. Ensure your hostname resolves to public iface IP - (for testing)
> edit
> >> > /etc/hosts to resolve your domain name to 192.168.56.50
> >> > 3. Set correct hostname/ip in mesos configuration - see Nikolaos
> answer
> >> >
> >>
> >> Cheers,
> >> Aaron
> >>
> >> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
> >> <iu...@typesafe.com> wrote:
> >> > Hi Aaron,
> >> >
> >> > I never had to use that variable. What is it for?
> >> >
> >> > On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
> >> >>
> >> >> In going through running various Spark jobs, both Spark 1.5.2 and the
> >> >> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
> >> >> noticed that is in order to run the Spark shells (both python and
> >> >> scala), we needed to set the LIBPROCESS_IP environment variable
> before
> >> >> running.
> >> >>
> >> >> Was curious if the Spark on Mesos docs should be updated, under the
> >> >> Client Mode section, to include setting this environment variable?
> >> >>
> >> >> Cheers
> >> >> Aaron
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> >> For additional commands, e-mail: dev-help@spark.apache.org
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > --
> >> > Iulian Dragos
> >> >
> >> > ------
> >> > Reactive Apps on the JVM
> >> > www.typesafe.com
> >> >
> >
> >
> >
> >
> > --
> >
> > --
> > Iulian Dragos
> >
> > ------
> > Reactive Apps on the JVM
> > www.typesafe.com
> >
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Aaron <aa...@gmail.com>.
Wrt to PR, sure, let me update the documentation, i'll send it out
shortly. My Fork is on Github..is the PR from there ok?
Cheers,
Aaron
On Wed, Dec 16, 2015 at 11:33 AM, Timothy Chen <tn...@gmail.com> wrote:
> Yes if want to manually override what IP to use to be contacted by the
> master you can set LIPROCESS_IP and LIBPROCESS_PORT.
>
> It is a Mesos specific settings. We can definitely update the docs.
>
> Note that in the future as we move to use the new Mesos Http API these
> configurations won't be needed (also libmesos!).
>
> Tim
>
> On Dec 16, 2015, at 8:09 AM, Iulian Dragoș <iu...@typesafe.com>
> wrote:
>
> LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a
> Mesos-specific setting.
>
> Have you tried setting SPARK_LOCAL_IP?
>
> On Wed, Dec 16, 2015 at 5:07 PM, Aaron <aa...@gmail.com> wrote:
>>
>> Found this thread that talked about it to help understand it better:
>>
>>
>> https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
>>
>> >
>> > When you run Spark on Mesos it needs to run
>> >
>> > spark driver
>> > mesos scheduler
>> >
>> > and both need to be visible to outside world on public iface IP
>> >
>> > you need to tell Spark and Mesos on which interface to bind - by default
>> > they resolve node hostname to ip - this is loopback address in your case
>> >
>> > Possible solutions - on slave node with public IP 192.168.56.50
>> >
>> > 1. Set
>> >
>> > export LIBPROCESS_IP=192.168.56.50
>> > export SPARK_LOCAL_IP=192.168.56.50
>> >
>> > 2. Ensure your hostname resolves to public iface IP - (for testing) edit
>> > /etc/hosts to resolve your domain name to 192.168.56.50
>> > 3. Set correct hostname/ip in mesos configuration - see Nikolaos answer
>> >
>>
>> Cheers,
>> Aaron
>>
>> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
>> <iu...@typesafe.com> wrote:
>> > Hi Aaron,
>> >
>> > I never had to use that variable. What is it for?
>> >
>> > On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
>> >>
>> >> In going through running various Spark jobs, both Spark 1.5.2 and the
>> >> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
>> >> noticed that is in order to run the Spark shells (both python and
>> >> scala), we needed to set the LIBPROCESS_IP environment variable before
>> >> running.
>> >>
>> >> Was curious if the Spark on Mesos docs should be updated, under the
>> >> Client Mode section, to include setting this environment variable?
>> >>
>> >> Cheers
>> >> Aaron
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> For additional commands, e-mail: dev-help@spark.apache.org
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > --
>> > Iulian Dragos
>> >
>> > ------
>> > Reactive Apps on the JVM
>> > www.typesafe.com
>> >
>
>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set for client mode
Posted by Timothy Chen <tn...@gmail.com>.
Yes if want to manually override what IP to use to be contacted by the master you can set LIPROCESS_IP and LIBPROCESS_PORT.
It is a Mesos specific settings. We can definitely update the docs.
Note that in the future as we move to use the new Mesos Http API these configurations won't be needed (also libmesos!).
Tim
> On Dec 16, 2015, at 8:09 AM, Iulian Dragoș <iu...@typesafe.com> wrote:
>
> LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a Mesos-specific setting.
>
> Have you tried setting SPARK_LOCAL_IP?
>
>> On Wed, Dec 16, 2015 at 5:07 PM, Aaron <aa...@gmail.com> wrote:
>> Found this thread that talked about it to help understand it better:
>>
>> https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
>>
>> >
>> > When you run Spark on Mesos it needs to run
>> >
>> > spark driver
>> > mesos scheduler
>> >
>> > and both need to be visible to outside world on public iface IP
>> >
>> > you need to tell Spark and Mesos on which interface to bind - by default
>> > they resolve node hostname to ip - this is loopback address in your case
>> >
>> > Possible solutions - on slave node with public IP 192.168.56.50
>> >
>> > 1. Set
>> >
>> > export LIBPROCESS_IP=192.168.56.50
>> > export SPARK_LOCAL_IP=192.168.56.50
>> >
>> > 2. Ensure your hostname resolves to public iface IP - (for testing) edit
>> > /etc/hosts to resolve your domain name to 192.168.56.50
>> > 3. Set correct hostname/ip in mesos configuration - see Nikolaos answer
>> >
>>
>> Cheers,
>> Aaron
>>
>> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
>> <iu...@typesafe.com> wrote:
>> > Hi Aaron,
>> >
>> > I never had to use that variable. What is it for?
>> >
>> > On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
>> >>
>> >> In going through running various Spark jobs, both Spark 1.5.2 and the
>> >> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
>> >> noticed that is in order to run the Spark shells (both python and
>> >> scala), we needed to set the LIBPROCESS_IP environment variable before
>> >> running.
>> >>
>> >> Was curious if the Spark on Mesos docs should be updated, under the
>> >> Client Mode section, to include setting this environment variable?
>> >>
>> >> Cheers
>> >> Aaron
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> For additional commands, e-mail: dev-help@spark.apache.org
>> >>
>> >
>> >
>> >
>> > --
>> >
>> > --
>> > Iulian Dragos
>> >
>> > ------
>> > Reactive Apps on the JVM
>> > www.typesafe.com
>> >
>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Iulian Dragoș <iu...@typesafe.com>.
LIBPROCESS_IP has zero hits in the Spark code base. This seems to be a
Mesos-specific setting.
Have you tried setting SPARK_LOCAL_IP?
On Wed, Dec 16, 2015 at 5:07 PM, Aaron <aa...@gmail.com> wrote:
> Found this thread that talked about it to help understand it better:
>
>
> https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
>
> >
> > When you run Spark on Mesos it needs to run
> >
> > spark driver
> > mesos scheduler
> >
> > and both need to be visible to outside world on public iface IP
> >
> > you need to tell Spark and Mesos on which interface to bind - by default
> > they resolve node hostname to ip - this is loopback address in your case
> >
> > Possible solutions - on slave node with public IP 192.168.56.50
> >
> > 1. Set
> >
> > export LIBPROCESS_IP=192.168.56.50
> > export SPARK_LOCAL_IP=192.168.56.50
> >
> > 2. Ensure your hostname resolves to public iface IP - (for testing) edit
> > /etc/hosts to resolve your domain name to 192.168.56.50
> > 3. Set correct hostname/ip in mesos configuration - see Nikolaos answer
> >
>
> Cheers,
> Aaron
>
> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
> <iu...@typesafe.com> wrote:
> > Hi Aaron,
> >
> > I never had to use that variable. What is it for?
> >
> > On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
> >>
> >> In going through running various Spark jobs, both Spark 1.5.2 and the
> >> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
> >> noticed that is in order to run the Spark shells (both python and
> >> scala), we needed to set the LIBPROCESS_IP environment variable before
> >> running.
> >>
> >> Was curious if the Spark on Mesos docs should be updated, under the
> >> Client Mode section, to include setting this environment variable?
> >>
> >> Cheers
> >> Aaron
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: dev-help@spark.apache.org
> >>
> >
> >
> >
> > --
> >
> > --
> > Iulian Dragos
> >
> > ------
> > Reactive Apps on the JVM
> > www.typesafe.com
> >
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Iulian Dragoș <iu...@typesafe.com>.
Sure, documenting this would be great, I just wanted to understand the
context. There is a related ticket: SPARK-5488
<https://issues.apache.org/jira/browse/SPARK-5488>.
Would you mind opening a PR?
On Wed, Dec 16, 2015 at 5:11 PM, Aaron <aa...@gmail.com> wrote:
> Basically, my hostname doesn't resolve to an "accessible" IP
> address...which isn't a big deal, I normally set SPARK_LOCAL_IP when I
> am doing things on a YARN cluster. But, we've moved to a Mesos
> Cluster recently, and had to track down when it wasn't working...I
> assumed (badly obviously) that setting SPARK_LOCAL_IP was
> sufficient...need to tell the Mesos scheduler as well I guess.
>
> Not sure if you if would be a good idea to put in actual code,
> something like: "when SPARK_LCOAL_IP is set, and using mesos:// as
> the master..set LIBPROCESS_IP," but, some kind of documentation about
> this possible issue would have saved me some time.
>
> Cheers,
> Aaron
>
> On Wed, Dec 16, 2015 at 11:07 AM, Aaron <aa...@gmail.com> wrote:
> > Found this thread that talked about it to help understand it better:
> >
> >
> https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
> >
> >>
> >> When you run Spark on Mesos it needs to run
> >>
> >> spark driver
> >> mesos scheduler
> >>
> >> and both need to be visible to outside world on public iface IP
> >>
> >> you need to tell Spark and Mesos on which interface to bind - by default
> >> they resolve node hostname to ip - this is loopback address in your case
> >>
> >> Possible solutions - on slave node with public IP 192.168.56.50
> >>
> >> 1. Set
> >>
> >> export LIBPROCESS_IP=192.168.56.50
> >> export SPARK_LOCAL_IP=192.168.56.50
> >>
> >> 2. Ensure your hostname resolves to public iface IP - (for testing) edit
> >> /etc/hosts to resolve your domain name to 192.168.56.50
> >> 3. Set correct hostname/ip in mesos configuration - see Nikolaos answer
> >>
> >
> > Cheers,
> > Aaron
> >
> > On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
> > <iu...@typesafe.com> wrote:
> >> Hi Aaron,
> >>
> >> I never had to use that variable. What is it for?
> >>
> >> On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
> >>>
> >>> In going through running various Spark jobs, both Spark 1.5.2 and the
> >>> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
> >>> noticed that is in order to run the Spark shells (both python and
> >>> scala), we needed to set the LIBPROCESS_IP environment variable before
> >>> running.
> >>>
> >>> Was curious if the Spark on Mesos docs should be updated, under the
> >>> Client Mode section, to include setting this environment variable?
> >>>
> >>> Cheers
> >>> Aaron
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> --
> >> Iulian Dragos
> >>
> >> ------
> >> Reactive Apps on the JVM
> >> www.typesafe.com
> >>
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Aaron <aa...@gmail.com>.
Basically, my hostname doesn't resolve to an "accessible" IP
address...which isn't a big deal, I normally set SPARK_LOCAL_IP when I
am doing things on a YARN cluster. But, we've moved to a Mesos
Cluster recently, and had to track down when it wasn't working...I
assumed (badly obviously) that setting SPARK_LOCAL_IP was
sufficient...need to tell the Mesos scheduler as well I guess.
Not sure if you if would be a good idea to put in actual code,
something like: "when SPARK_LCOAL_IP is set, and using mesos:// as
the master..set LIBPROCESS_IP," but, some kind of documentation about
this possible issue would have saved me some time.
Cheers,
Aaron
On Wed, Dec 16, 2015 at 11:07 AM, Aaron <aa...@gmail.com> wrote:
> Found this thread that talked about it to help understand it better:
>
> https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
>
>>
>> When you run Spark on Mesos it needs to run
>>
>> spark driver
>> mesos scheduler
>>
>> and both need to be visible to outside world on public iface IP
>>
>> you need to tell Spark and Mesos on which interface to bind - by default
>> they resolve node hostname to ip - this is loopback address in your case
>>
>> Possible solutions - on slave node with public IP 192.168.56.50
>>
>> 1. Set
>>
>> export LIBPROCESS_IP=192.168.56.50
>> export SPARK_LOCAL_IP=192.168.56.50
>>
>> 2. Ensure your hostname resolves to public iface IP - (for testing) edit
>> /etc/hosts to resolve your domain name to 192.168.56.50
>> 3. Set correct hostname/ip in mesos configuration - see Nikolaos answer
>>
>
> Cheers,
> Aaron
>
> On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
> <iu...@typesafe.com> wrote:
>> Hi Aaron,
>>
>> I never had to use that variable. What is it for?
>>
>> On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
>>>
>>> In going through running various Spark jobs, both Spark 1.5.2 and the
>>> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
>>> noticed that is in order to run the Spark shells (both python and
>>> scala), we needed to set the LIBPROCESS_IP environment variable before
>>> running.
>>>
>>> Was curious if the Spark on Mesos docs should be updated, under the
>>> Client Mode section, to include setting this environment variable?
>>>
>>> Cheers
>>> Aaron
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>
>>
>>
>> --
>>
>> --
>> Iulian Dragos
>>
>> ------
>> Reactive Apps on the JVM
>> www.typesafe.com
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Aaron <aa...@gmail.com>.
Found this thread that talked about it to help understand it better:
https://mail-archives.apache.org/mod_mbox/mesos-user/201507.mbox/%3CCAJQ68qf9pEJgNWOmasM2DQchYaxPcAovnFKFgGgxXpZJ2JOAfQ@mail.gmail.com%3E
>
> When you run Spark on Mesos it needs to run
>
> spark driver
> mesos scheduler
>
> and both need to be visible to outside world on public iface IP
>
> you need to tell Spark and Mesos on which interface to bind - by default
> they resolve node hostname to ip - this is loopback address in your case
>
> Possible solutions - on slave node with public IP 192.168.56.50
>
> 1. Set
>
> export LIBPROCESS_IP=192.168.56.50
> export SPARK_LOCAL_IP=192.168.56.50
>
> 2. Ensure your hostname resolves to public iface IP - (for testing) edit
> /etc/hosts to resolve your domain name to 192.168.56.50
> 3. Set correct hostname/ip in mesos configuration - see Nikolaos answer
>
Cheers,
Aaron
On Wed, Dec 16, 2015 at 11:00 AM, Iulian Dragoș
<iu...@typesafe.com> wrote:
> Hi Aaron,
>
> I never had to use that variable. What is it for?
>
> On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
>>
>> In going through running various Spark jobs, both Spark 1.5.2 and the
>> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
>> noticed that is in order to run the Spark shells (both python and
>> scala), we needed to set the LIBPROCESS_IP environment variable before
>> running.
>>
>> Was curious if the Spark on Mesos docs should be updated, under the
>> Client Mode section, to include setting this environment variable?
>>
>> Cheers
>> Aaron
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>
>
>
> --
>
> --
> Iulian Dragos
>
> ------
> Reactive Apps on the JVM
> www.typesafe.com
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Iulian Dragoș <iu...@typesafe.com>.
Hi Aaron,
I never had to use that variable. What is it for?
On Wed, Dec 16, 2015 at 2:00 PM, Aaron <aa...@gmail.com> wrote:
> In going through running various Spark jobs, both Spark 1.5.2 and the
> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
> noticed that is in order to run the Spark shells (both python and
> scala), we needed to set the LIBPROCESS_IP environment variable before
> running.
>
> Was curious if the Spark on Mesos docs should be updated, under the
> Client Mode section, to include setting this environment variable?
>
> Cheers
> Aaron
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
--
--
Iulian Dragos
------
Reactive Apps on the JVM
www.typesafe.com
Re: Update to Spar Mesos docs possibly? LIBPROCESS_IP needs to be set
for client mode
Posted by Aaron <aa...@gmail.com>.
The more I read/look into this..it's because the Spark Mesos Scheduler
resolves to something can't be reached (e.g. localhost). So, maybe
this just needs to be added to the docs, if your host cannot or does
not resolve to the IP address you want. Maybe just a footnote or
something?
On Wed, Dec 16, 2015 at 8:00 AM, Aaron <aa...@gmail.com> wrote:
> In going through running various Spark jobs, both Spark 1.5.2 and the
> new Spark 1.6 SNAPSHOTs, on a Mesos cluster (currently 0.25), we
> noticed that is in order to run the Spark shells (both python and
> scala), we needed to set the LIBPROCESS_IP environment variable before
> running.
>
> Was curious if the Spark on Mesos docs should be updated, under the
> Client Mode section, to include setting this environment variable?
>
> Cheers
> Aaron
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org