You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2015/04/11 05:24:58 UTC

wait time between start master and start slaves

Check this out
<https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33>
(from spark-ec2):

# Start Master$BIN_FOLDER/start-master.sh
# Pause
sleep 20
# Start Workers$BIN_FOLDER/start-slaves.sh

I know this was probably done defensively, but is there a more direct way
to know when the master is ready?

Nick
​

Re: wait time between start master and start slaves

Posted by Nicholas Chammas <ni...@gmail.com>.
For the record, this is what I came up with (ignoring the configurable port
for now):

spark/sbin/start-master.sh

master_ui_response_code=0
while [ "$master_ui_response_code" -ne 200 ]; do
    sleep 1
    master_ui_response_code="$(
        curl --head --silent --output /dev/null \
             --write-out "%{http_code}" localhost:8080
    )"done

spark/sbin/start-slaves.sh

Turns out that the master typically takes 3-4 seconds to come up.

That’s 15 seconds saved. Hurray for yak shaving!

Nick
​

On Sun, Apr 12, 2015 at 5:56 PM Nicholas Chammas <ni...@gmail.com>
wrote:

> Oh, good point. So I guess I should be able to query the master via code
> like this before any slaves are started.
>
> On Sat, Apr 11, 2015 at 7:52 PM Ted Yu <yu...@gmail.com> wrote:
>
>> From SparkUI.scala :
>>
>>   def getUIPort(conf: SparkConf): Int = {
>>     conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT)
>>   }
>> Better retrieve effective UI port before probing.
>>
>> Cheers
>>
>> On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> So basically, to tell if the master is ready to accept slaves, just poll
>>> http://master-node:4040 for an HTTP 200 response?
>>> ​
>>>
>>> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
>>> shivaram@eecs.berkeley.edu> wrote:
>>>
>>> > Yeah from what I remember it was set defensively. I don't know of a
>>> good
>>> > way to check if the master is up though. I guess we could poll the
>>> Master
>>> > Web UI and see if we get a 200/ok response
>>> >
>>> > Shivaram
>>> >
>>> > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
>>> > nicholas.chammas@gmail.com> wrote:
>>> >
>>> >> Check this out
>>> >> <
>>> >>
>>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>>> >> >
>>> >> (from spark-ec2):
>>> >>
>>> >> # Start Master$BIN_FOLDER/start-master.sh
>>> >
>>> >
>>> >> # Pause
>>> >> sleep 20
>>> >> # Start Workers$BIN_FOLDER/start-slaves.sh
>>> >>
>>> >> I know this was probably done defensively, but is there a more direct
>>> way
>>> >> to know when the master is ready?
>>> >>
>>> >> Nick
>>> >> ​
>>> >>
>>> >
>>>
>>
>>

Re: wait time between start master and start slaves

Posted by Nicholas Chammas <ni...@gmail.com>.
Oh, good point. So I guess I should be able to query the master via code
like this before any slaves are started.

On Sat, Apr 11, 2015 at 7:52 PM Ted Yu <yu...@gmail.com> wrote:

> From SparkUI.scala :
>
>   def getUIPort(conf: SparkConf): Int = {
>     conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT)
>   }
> Better retrieve effective UI port before probing.
>
> Cheers
>
> On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> So basically, to tell if the master is ready to accept slaves, just poll
>> http://master-node:4040 for an HTTP 200 response?
>> ​
>>
>> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
>> shivaram@eecs.berkeley.edu> wrote:
>>
>> > Yeah from what I remember it was set defensively. I don't know of a good
>> > way to check if the master is up though. I guess we could poll the
>> Master
>> > Web UI and see if we get a 200/ok response
>> >
>> > Shivaram
>> >
>> > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
>> > nicholas.chammas@gmail.com> wrote:
>> >
>> >> Check this out
>> >> <
>> >>
>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>> >> >
>> >> (from spark-ec2):
>> >>
>> >> # Start Master$BIN_FOLDER/start-master.sh
>> >
>> >
>> >> # Pause
>> >> sleep 20
>> >> # Start Workers$BIN_FOLDER/start-slaves.sh
>> >>
>> >> I know this was probably done defensively, but is there a more direct
>> way
>> >> to know when the master is ready?
>> >>
>> >> Nick
>> >> ​
>> >>
>> >
>>
>
>

Re: wait time between start master and start slaves

Posted by Ted Yu <yu...@gmail.com>.
>From SparkUI.scala :

  def getUIPort(conf: SparkConf): Int = {
    conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT)
  }
Better retrieve effective UI port before probing.

Cheers

On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> So basically, to tell if the master is ready to accept slaves, just poll
> http://master-node:4040 for an HTTP 200 response?
> ​
>
> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
> > Yeah from what I remember it was set defensively. I don't know of a good
> > way to check if the master is up though. I guess we could poll the Master
> > Web UI and see if we get a 200/ok response
> >
> > Shivaram
> >
> > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
> > nicholas.chammas@gmail.com> wrote:
> >
> >> Check this out
> >> <
> >>
> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
> >> >
> >> (from spark-ec2):
> >>
> >> # Start Master$BIN_FOLDER/start-master.sh
> >
> >
> >> # Pause
> >> sleep 20
> >> # Start Workers$BIN_FOLDER/start-slaves.sh
> >>
> >> I know this was probably done defensively, but is there a more direct
> way
> >> to know when the master is ready?
> >>
> >> Nick
> >> ​
> >>
> >
>

Re: wait time between start master and start slaves

Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
Yeah thats the best I can think ok -- Not sure if there is a better way to
do it.

On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> So basically, to tell if the master is ready to accept slaves, just poll
> http://master-node:4040 for an HTTP 200 response?
> ​
>
> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
>> Yeah from what I remember it was set defensively. I don't know of a good
>> way to check if the master is up though. I guess we could poll the Master
>> Web UI and see if we get a 200/ok response
>>
>> Shivaram
>>
>> On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> Check this out
>>> <
>>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>>> >
>>> (from spark-ec2):
>>>
>>> # Start Master$BIN_FOLDER/start-master.sh
>>
>>
>>> # Pause
>>> sleep 20
>>> # Start Workers$BIN_FOLDER/start-slaves.sh
>>>
>>> I know this was probably done defensively, but is there a more direct way
>>> to know when the master is ready?
>>>
>>> Nick
>>> ​
>>>
>>

Re: wait time between start master and start slaves

Posted by Nicholas Chammas <ni...@gmail.com>.
So basically, to tell if the master is ready to accept slaves, just poll
http://master-node:4040 for an HTTP 200 response?
​

On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:

> Yeah from what I remember it was set defensively. I don't know of a good
> way to check if the master is up though. I guess we could poll the Master
> Web UI and see if we get a 200/ok response
>
> Shivaram
>
> On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Check this out
>> <
>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>> >
>> (from spark-ec2):
>>
>> # Start Master$BIN_FOLDER/start-master.sh
>
>
>> # Pause
>> sleep 20
>> # Start Workers$BIN_FOLDER/start-slaves.sh
>>
>> I know this was probably done defensively, but is there a more direct way
>> to know when the master is ready?
>>
>> Nick
>> ​
>>
>

Re: wait time between start master and start slaves

Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
Yeah from what I remember it was set defensively. I don't know of a good
way to check if the master is up though. I guess we could poll the Master
Web UI and see if we get a 200/ok response

Shivaram

On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:

> Check this out
> <
> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
> >
> (from spark-ec2):
>
> # Start Master$BIN_FOLDER/start-master.sh
> # Pause
> sleep 20
> # Start Workers$BIN_FOLDER/start-slaves.sh
>
> I know this was probably done defensively, but is there a more direct way
> to know when the master is ready?
>
> Nick
> ​
>