You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2015/10/15 04:10:41 UTC
SPARK_MASTER_IP actually expects a DNS name, not IP address
I’m setting the Spark master address via the SPARK_MASTER_IP environment
variable in spark-env.sh, like spark-ec2 does
<https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13>
.
The funny thing is that Spark seems to accept this only if the value of
SPARK_MASTER_IP is a DNS name and not an IP address.
When I provide an IP address, I get errors in the log when starting the
master:
15/10/15 01:47:31 ERROR NettyTransport: failed to bind to
/54.210.XX.XX:7077, shutting down Netty transport
(XX is my redaction of the full IP address.)
Am I misunderstanding something about how to use this environment variable?
The spark-env.sh template indicates that either an IP address or a hostname
should work
<https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49>,
but my testing shows that only hostnames work.
Nick
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Robert Dodier <ro...@gmail.com>.
Nicholas,
FWIW the --ip option seems to have been deprecated in commit d90d2af1,
but that was a pretty big commit, lots of other stuff changed, and there
isn't any hint in the log message as to the reason for changing --ip.
best,
Robert Dodier
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Nicholas Chammas <ni...@gmail.com>.
Good catches, Robert.
I had actually typed up a draft email a couple of days ago citing those
same two blocks of code. I deleted it when I realized like you that the
snippets did not explain why IP addresses weren’t working.
Something seems wrong here, but I’m not sure what exactly. Maybe this is a
documentation bug or missing deprecation warning.
For example, this line
<https://github.com/apache/spark/blob/a337c235a12d4ea6a7d6db457acc6b32f1915241/core/src/main/scala/org/apache/spark/deploy/master/MasterArguments.scala#L93>
seems to back up your finding that SPARK_MASTER_IP is deprecated (since the
--ip it maps to is deprecated), but no warnings are displayed and the docs
make no mention of this being deprecated.
private def printUsageAndExit(exitCode: Int) {
// scalastyle:off println
System.err.println(
"Usage: Master [options]\n" +
"\n" +
"Options:\n" +
" -i HOST, --ip HOST Hostname to listen on (deprecated,
please use --host or -h) \n" +
" -h HOST, --host HOST Hostname to listen on\n" +
" -p PORT, --port PORT Port to listen on (default: 7077)\n" +
" --webui-port PORT Port for web UI (default: 8080)\n" +
" --properties-file FILE Path to a custom Spark properties file.\n" +
" Default is conf/spark-defaults.conf.")
// scalastyle:on println
System.exit(exitCode)
}
Hopefully someone with better knowledge of this code can explain what’s
going on.
I’m beginning to think SPARK_MASTER_IP is straight up deprecated in favor
of SPARK_MASTER_HOST, but a warning needs to be actually thrown here in the
code
<https://github.com/apache/spark/blob/a337c235a12d4ea6a7d6db457acc6b32f1915241/core/src/main/scala/org/apache/spark/deploy/master/MasterArguments.scala#L52-L56>
and the template
<https://github.com/apache/spark/blob/a337c235a12d4ea6a7d6db457acc6b32f1915241/conf/spark-env.sh.template#L49>
and docs need to be updated.
This code hasn’t been touched much since Spark’s genesis, so I’m not
expecting anyone to know off the top of their head whether this is wrong or
right. Perhaps I should just open a PR and take it from there.
Nick
On Sat, Oct 17, 2015 at 11:21 PM Robert Dodier <ro...@gmail.com>
wrote:
> Nicholas Chammas wrote
> > The funny thing is that Spark seems to accept this only if the value of
> > SPARK_MASTER_IP is a DNS name and not an IP address.
> >
> > When I provide an IP address, I get errors in the log when starting the
> > master:
> >
> > 15/10/15 01:47:31 ERROR NettyTransport: failed to bind to
> > /54.210.XX.XX:7077, shutting down Netty transport
>
> A couple of things. (1) That log message appears to originate at line 434
> of
> NettyTransport.scala.
> (
> https://github.com/akka/akka/blob/master/akka-remote/src/main/scala/akka/remote/transport/netty/NettyTransport.scala
> )
> It appears the exception is rethrown; is it caught somewhere else so we can
> see what the actual error was that triggered the log message? I don't see
> anything obvious in the code.
>
> (2) sbin/start-master.sh executes something.Master with --ip
> SPARK_MASTER_IP, which calls something.MasterArguments to handle its
> arguments, which says:
>
> case ("--ip" | "-i") :: value :: tail =>
> Utils.checkHost(value, "ip no longer supported, please use hostname
> " + value)
> host = value
> parse(tail)
>
> case ("--host" | "-h") :: value :: tail =>
> Utils.checkHost(value, "Please use hostname " + value)
> host = value
> parse(tail)
>
> So it would appear that the intent is that numerical IP addresses are
> disallowed, however, Utils.checkHost says:
>
> def checkHost(host: String, message: String = "") {
> assert(host.indexOf(':') == -1, message)
> }
>
> which accepts numerical IP addresses just fine. Is there some other test
> that should be applied in MasterArguments? or maybe checkHost should be
> looking for some other pattern? Is it possible that MasterArguments was
> changed to disallow --ip without propagating that backwards into any
> scripts
> that call it?
>
> Hope this helps in some way.
>
> Robert Dodier
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/SPARK-MASTER-IP-actually-expects-a-DNS-name-not-IP-address-tp14613p14665.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Robert Dodier <ro...@gmail.com>.
Nicholas Chammas wrote
> The funny thing is that Spark seems to accept this only if the value of
> SPARK_MASTER_IP is a DNS name and not an IP address.
>
> When I provide an IP address, I get errors in the log when starting the
> master:
>
> 15/10/15 01:47:31 ERROR NettyTransport: failed to bind to
> /54.210.XX.XX:7077, shutting down Netty transport
A couple of things. (1) That log message appears to originate at line 434 of
NettyTransport.scala.
(https://github.com/akka/akka/blob/master/akka-remote/src/main/scala/akka/remote/transport/netty/NettyTransport.scala)
It appears the exception is rethrown; is it caught somewhere else so we can
see what the actual error was that triggered the log message? I don't see
anything obvious in the code.
(2) sbin/start-master.sh executes something.Master with --ip
SPARK_MASTER_IP, which calls something.MasterArguments to handle its
arguments, which says:
case ("--ip" | "-i") :: value :: tail =>
Utils.checkHost(value, "ip no longer supported, please use hostname
" + value)
host = value
parse(tail)
case ("--host" | "-h") :: value :: tail =>
Utils.checkHost(value, "Please use hostname " + value)
host = value
parse(tail)
So it would appear that the intent is that numerical IP addresses are
disallowed, however, Utils.checkHost says:
def checkHost(host: String, message: String = "") {
assert(host.indexOf(':') == -1, message)
}
which accepts numerical IP addresses just fine. Is there some other test
that should be applied in MasterArguments? or maybe checkHost should be
looking for some other pattern? Is it possible that MasterArguments was
changed to disallow --ip without propagating that backwards into any scripts
that call it?
Hope this helps in some way.
Robert Dodier
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/SPARK-MASTER-IP-actually-expects-a-DNS-name-not-IP-address-tp14613p14665.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Nicholas Chammas <ni...@gmail.com>.
JB,
I am using spark-env.sh to define the master address instead of using
spark-defaults.conf.
I understand that that should work, and indeed it does, but only if
SPARK_MASTER_IP is set to a DNS name and not an IP address.
Perhaps I'm misunderstanding these configuration methods...
Nick
On Fri, Oct 16, 2015 at 12:05 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Nick,
>
> there's the Spark master defined in conf/spark-defaults.conf and the -h
> option that you can provide to sbin/start-master.sh script.
>
> Did you try:
>
> sbin/start-master.sh -h xxx.xxx.xxx.xxx
>
> and then use the IP when you start the slaves:
>
> sbin/start-slave.sh spark://xxx.xxx.xxx.xxx.7077
>
> ?
>
> Regards
> JB
>
> On 10/16/2015 06:01 PM, Nicholas Chammas wrote:
> > I'd look into tracing a possible bug here, but I'm not sure where to
> > look. Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not
> > show it being used in any place directly by Spark
> > <https://github.com/apache/spark/search?utf8=%E2%9C%93&q=SPARK_MASTER_IP
> >.
> >
> > Clearly, Spark is using this environment variable (otherwise I wouldn't
> > see the behavior described in my first email), but I can't see where.
> >
> > Can someone give me a pointer?
> >
> > Nick
> >
> > On Thu, Oct 15, 2015 at 12:37 AM Ted Yu <yuzhihong@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Some old bits:
> >
> >
> http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
> >
> http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
> >
> > FYI
> >
> > On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas
> > <nicholas.chammas@gmail.com <ma...@gmail.com>>
> wrote:
> >
> > I’m setting the Spark master address via the |SPARK_MASTER_IP|
> > environment variable in |spark-env.sh|, like spark-ec2 does
> > <
> https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13
> >.
> >
> > The funny thing is that Spark seems to accept this only if the
> > value of |SPARK_MASTER_IP| is a DNS name and not an IP address.
> >
> > When I provide an IP address, I get errors in the log when
> > starting the master:
> >
> > |15/10/15 01:47:31 ERROR NettyTransport: failed to bind to
> > /54.210.XX.XX:7077, shutting down Netty transport |
> >
> > (XX is my redaction of the full IP address.)
> >
> > Am I misunderstanding something about how to use this
> > environment variable?
> >
> > The spark-env.sh template indicates that either an IP address or
> > a hostname should work
> > <
> https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49
> >,
> > but my testing shows that only hostnames work.
> >
> > Nick
> >
> >
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Nick,
there's the Spark master defined in conf/spark-defaults.conf and the -h
option that you can provide to sbin/start-master.sh script.
Did you try:
sbin/start-master.sh -h xxx.xxx.xxx.xxx
and then use the IP when you start the slaves:
sbin/start-slave.sh spark://xxx.xxx.xxx.xxx.7077
?
Regards
JB
On 10/16/2015 06:01 PM, Nicholas Chammas wrote:
> I'd look into tracing a possible bug here, but I'm not sure where to
> look. Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not
> show it being used in any place directly by Spark
> <https://github.com/apache/spark/search?utf8=%E2%9C%93&q=SPARK_MASTER_IP>.
>
> Clearly, Spark is using this environment variable (otherwise I wouldn't
> see the behavior described in my first email), but I can't see where.
>
> Can someone give me a pointer?
>
> Nick
>
> On Thu, Oct 15, 2015 at 12:37 AM Ted Yu <yuzhihong@gmail.com
> <ma...@gmail.com>> wrote:
>
> Some old bits:
>
> http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
> http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
>
> FYI
>
> On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas
> <nicholas.chammas@gmail.com <ma...@gmail.com>> wrote:
>
> I’m setting the Spark master address via the |SPARK_MASTER_IP|
> environment variable in |spark-env.sh|, like spark-ec2 does
> <https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13>.
>
> The funny thing is that Spark seems to accept this only if the
> value of |SPARK_MASTER_IP| is a DNS name and not an IP address.
>
> When I provide an IP address, I get errors in the log when
> starting the master:
>
> |15/10/15 01:47:31 ERROR NettyTransport: failed to bind to
> /54.210.XX.XX:7077, shutting down Netty transport |
>
> (XX is my redaction of the full IP address.)
>
> Am I misunderstanding something about how to use this
> environment variable?
>
> The spark-env.sh template indicates that either an IP address or
> a hostname should work
> <https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49>,
> but my testing shows that only hostnames work.
>
> Nick
>
>
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Ted Yu <yu...@gmail.com>.
if [ "$SPARK_MASTER_IP" = "" ]; then
SPARK_MASTER_IP=`hostname`
--ip $SPARK_MASTER_IP --port $SPARK_MASTER_PORT --webui-port
$SPARK_MASTER_WEBUI_PORT \
"$sbin"/../tachyon/bin/tachyon bootstrap-conf $SPARK_MASTER_IP
./sbin/start-master.sh
if [ "$SPARK_MASTER_IP" = "" ]; then
SPARK_MASTER_IP="`hostname`"
"$sbin/slaves.sh" cd "$SPARK_HOME" \; "$sbin"/../tachyon/bin/tachyon
bootstrap-conf "$SPARK_MASTER_IP"
"$sbin/slaves.sh" cd "$SPARK_HOME" \; "$sbin/start-slave.sh"
"spark://$SPARK_MASTER_IP:$SPARK_MASTER_PORT"
./sbin/start-slaves.sh
On Fri, Oct 16, 2015 at 9:01 AM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> I'd look into tracing a possible bug here, but I'm not sure where to look.
> Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not show it
> being used in any place directly by Spark
> <https://github.com/apache/spark/search?utf8=%E2%9C%93&q=SPARK_MASTER_IP>.
>
> Clearly, Spark is using this environment variable (otherwise I wouldn't
> see the behavior described in my first email), but I can't see where.
>
> Can someone give me a pointer?
>
> Nick
>
> On Thu, Oct 15, 2015 at 12:37 AM Ted Yu <yu...@gmail.com> wrote:
>
>> Some old bits:
>>
>>
>> http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
>> http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
>>
>> FYI
>>
>> On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> I’m setting the Spark master address via the SPARK_MASTER_IP
>>> environment variable in spark-env.sh, like spark-ec2 does
>>> <https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13>
>>> .
>>>
>>> The funny thing is that Spark seems to accept this only if the value of
>>> SPARK_MASTER_IP is a DNS name and not an IP address.
>>>
>>> When I provide an IP address, I get errors in the log when starting the
>>> master:
>>>
>>> 15/10/15 01:47:31 ERROR NettyTransport: failed to bind to /54.210.XX.XX:7077, shutting down Netty transport
>>>
>>> (XX is my redaction of the full IP address.)
>>>
>>> Am I misunderstanding something about how to use this environment
>>> variable?
>>>
>>> The spark-env.sh template indicates that either an IP address or a
>>> hostname should work
>>> <https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49>,
>>> but my testing shows that only hostnames work.
>>>
>>> Nick
>>>
>>>
>>
>>
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Nicholas Chammas <ni...@gmail.com>.
Ah, my bad, I missed it
<https://github.com/apache/spark/blob/08698ee1d6f29b2c999416f18a074d5193cdacd5/sbin/start-master.sh#L58-L60>
since the GitHub search results preview only showed
<https://github.com/apache/spark/search?utf8=%E2%9C%93&q=SPARK_MASTER_IP>
the first hit from start-master.sh and not this part:
"$sbin"/spark-daemon.sh start org.apache.spark.deploy.master.Master 1 \
--ip $SPARK_MASTER_IP --port $SPARK_MASTER_PORT --webui-port
$SPARK_MASTER_WEBUI_PORT \
$ORIGINAL_ARGS
Same goes for some of the other sbin scripts.
Anyway, let’s take a closer look…
Nick
On Fri, Oct 16, 2015 at 12:05 PM Sean Owen <so...@cloudera.com> wrote:
> It's used in scripts like sbin/start-master.sh
>
> On Fri, Oct 16, 2015 at 5:01 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I'd look into tracing a possible bug here, but I'm not sure where to
>> look. Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not
>> show it being used in any place directly by Spark
>> <https://github.com/apache/spark/search?utf8=%E2%9C%93&q=SPARK_MASTER_IP>
>> .
>>
>> Clearly, Spark is using this environment variable (otherwise I wouldn't
>> see the behavior described in my first email), but I can't see where.
>>
>> Can someone give me a pointer?
>>
>> Nick
>>
>> On Thu, Oct 15, 2015 at 12:37 AM Ted Yu <yu...@gmail.com> wrote:
>>
>>> Some old bits:
>>>
>>>
>>> http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
>>> http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
>>>
>>> FYI
>>>
>>> On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas <
>>> nicholas.chammas@gmail.com> wrote:
>>>
>>>> I’m setting the Spark master address via the SPARK_MASTER_IP
>>>> environment variable in spark-env.sh, like spark-ec2 does
>>>> <https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13>
>>>> .
>>>>
>>>> The funny thing is that Spark seems to accept this only if the value of
>>>> SPARK_MASTER_IP is a DNS name and not an IP address.
>>>>
>>>> When I provide an IP address, I get errors in the log when starting the
>>>> master:
>>>>
>>>> 15/10/15 01:47:31 ERROR NettyTransport: failed to bind to /54.210.XX.XX:7077, shutting down Netty transport
>>>>
>>>> (XX is my redaction of the full IP address.)
>>>>
>>>> Am I misunderstanding something about how to use this environment
>>>> variable?
>>>>
>>>> The spark-env.sh template indicates that either an IP address or a
>>>> hostname should work
>>>> <https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49>,
>>>> but my testing shows that only hostnames work.
>>>>
>>>> Nick
>>>>
>>>>
>>>
>>>
>
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Sean Owen <so...@cloudera.com>.
It's used in scripts like sbin/start-master.sh
On Fri, Oct 16, 2015 at 5:01 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> I'd look into tracing a possible bug here, but I'm not sure where to look.
> Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not show it
> being used in any place directly by Spark
> <https://github.com/apache/spark/search?utf8=%E2%9C%93&q=SPARK_MASTER_IP>.
>
> Clearly, Spark is using this environment variable (otherwise I wouldn't
> see the behavior described in my first email), but I can't see where.
>
> Can someone give me a pointer?
>
> Nick
>
> On Thu, Oct 15, 2015 at 12:37 AM Ted Yu <yu...@gmail.com> wrote:
>
>> Some old bits:
>>
>>
>> http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
>> http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
>>
>> FYI
>>
>> On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> I’m setting the Spark master address via the SPARK_MASTER_IP
>>> environment variable in spark-env.sh, like spark-ec2 does
>>> <https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13>
>>> .
>>>
>>> The funny thing is that Spark seems to accept this only if the value of
>>> SPARK_MASTER_IP is a DNS name and not an IP address.
>>>
>>> When I provide an IP address, I get errors in the log when starting the
>>> master:
>>>
>>> 15/10/15 01:47:31 ERROR NettyTransport: failed to bind to /54.210.XX.XX:7077, shutting down Netty transport
>>>
>>> (XX is my redaction of the full IP address.)
>>>
>>> Am I misunderstanding something about how to use this environment
>>> variable?
>>>
>>> The spark-env.sh template indicates that either an IP address or a
>>> hostname should work
>>> <https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49>,
>>> but my testing shows that only hostnames work.
>>>
>>> Nick
>>>
>>>
>>
>>
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Nicholas Chammas <ni...@gmail.com>.
I'd look into tracing a possible bug here, but I'm not sure where to look.
Searching the codebase for `SPARK_MASTER_IP`, amazingly, does not show it
being used in any place directly by Spark
<https://github.com/apache/spark/search?utf8=%E2%9C%93&q=SPARK_MASTER_IP>.
Clearly, Spark is using this environment variable (otherwise I wouldn't see
the behavior described in my first email), but I can't see where.
Can someone give me a pointer?
Nick
On Thu, Oct 15, 2015 at 12:37 AM Ted Yu <yu...@gmail.com> wrote:
> Some old bits:
>
>
> http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
> http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
>
> FYI
>
> On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I’m setting the Spark master address via the SPARK_MASTER_IP environment
>> variable in spark-env.sh, like spark-ec2 does
>> <https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13>
>> .
>>
>> The funny thing is that Spark seems to accept this only if the value of
>> SPARK_MASTER_IP is a DNS name and not an IP address.
>>
>> When I provide an IP address, I get errors in the log when starting the
>> master:
>>
>> 15/10/15 01:47:31 ERROR NettyTransport: failed to bind to /54.210.XX.XX:7077, shutting down Netty transport
>>
>> (XX is my redaction of the full IP address.)
>>
>> Am I misunderstanding something about how to use this environment
>> variable?
>>
>> The spark-env.sh template indicates that either an IP address or a
>> hostname should work
>> <https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49>,
>> but my testing shows that only hostnames work.
>>
>> Nick
>>
>>
>
>
Re: SPARK_MASTER_IP actually expects a DNS name, not IP address
Posted by Ted Yu <yu...@gmail.com>.
Some old bits:
http://stackoverflow.com/questions/28162991/cant-run-spark-1-2-in-standalone-mode-on-mac
http://stackoverflow.com/questions/29412157/passing-hostname-to-netty
FYI
On Wed, Oct 14, 2015 at 7:10 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> I’m setting the Spark master address via the SPARK_MASTER_IP environment
> variable in spark-env.sh, like spark-ec2 does
> <https://github.com/amplab/spark-ec2/blob/a990752575cd8b0ab25731d7820a55c714798ec3/templates/root/spark/conf/spark-env.sh#L13>
> .
>
> The funny thing is that Spark seems to accept this only if the value of
> SPARK_MASTER_IP is a DNS name and not an IP address.
>
> When I provide an IP address, I get errors in the log when starting the
> master:
>
> 15/10/15 01:47:31 ERROR NettyTransport: failed to bind to /54.210.XX.XX:7077, shutting down Netty transport
>
> (XX is my redaction of the full IP address.)
>
> Am I misunderstanding something about how to use this environment variable?
>
> The spark-env.sh template indicates that either an IP address or a
> hostname should work
> <https://github.com/apache/spark/blob/4ace4f8a9c91beb21a0077e12b75637a4560a542/conf/spark-env.sh.template#L49>,
> but my testing shows that only hostnames work.
>
> Nick
>
>