You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2015/04/11 05:24:58 UTC
wait time between start master and start slaves
Check this out
<https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33>
(from spark-ec2):
# Start Master$BIN_FOLDER/start-master.sh
# Pause
sleep 20
# Start Workers$BIN_FOLDER/start-slaves.sh
I know this was probably done defensively, but is there a more direct way
to know when the master is ready?
Nick
Re: wait time between start master and start slaves
Posted by Nicholas Chammas <ni...@gmail.com>.
For the record, this is what I came up with (ignoring the configurable port
for now):
spark/sbin/start-master.sh
master_ui_response_code=0
while [ "$master_ui_response_code" -ne 200 ]; do
sleep 1
master_ui_response_code="$(
curl --head --silent --output /dev/null \
--write-out "%{http_code}" localhost:8080
)"done
spark/sbin/start-slaves.sh
Turns out that the master typically takes 3-4 seconds to come up.
That’s 15 seconds saved. Hurray for yak shaving!
Nick
On Sun, Apr 12, 2015 at 5:56 PM Nicholas Chammas <ni...@gmail.com>
wrote:
> Oh, good point. So I guess I should be able to query the master via code
> like this before any slaves are started.
>
> On Sat, Apr 11, 2015 at 7:52 PM Ted Yu <yu...@gmail.com> wrote:
>
>> From SparkUI.scala :
>>
>> def getUIPort(conf: SparkConf): Int = {
>> conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT)
>> }
>> Better retrieve effective UI port before probing.
>>
>> Cheers
>>
>> On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> So basically, to tell if the master is ready to accept slaves, just poll
>>> http://master-node:4040 for an HTTP 200 response?
>>>
>>>
>>> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
>>> shivaram@eecs.berkeley.edu> wrote:
>>>
>>> > Yeah from what I remember it was set defensively. I don't know of a
>>> good
>>> > way to check if the master is up though. I guess we could poll the
>>> Master
>>> > Web UI and see if we get a 200/ok response
>>> >
>>> > Shivaram
>>> >
>>> > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
>>> > nicholas.chammas@gmail.com> wrote:
>>> >
>>> >> Check this out
>>> >> <
>>> >>
>>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>>> >> >
>>> >> (from spark-ec2):
>>> >>
>>> >> # Start Master$BIN_FOLDER/start-master.sh
>>> >
>>> >
>>> >> # Pause
>>> >> sleep 20
>>> >> # Start Workers$BIN_FOLDER/start-slaves.sh
>>> >>
>>> >> I know this was probably done defensively, but is there a more direct
>>> way
>>> >> to know when the master is ready?
>>> >>
>>> >> Nick
>>> >>
>>> >>
>>> >
>>>
>>
>>
Re: wait time between start master and start slaves
Posted by Nicholas Chammas <ni...@gmail.com>.
Oh, good point. So I guess I should be able to query the master via code
like this before any slaves are started.
On Sat, Apr 11, 2015 at 7:52 PM Ted Yu <yu...@gmail.com> wrote:
> From SparkUI.scala :
>
> def getUIPort(conf: SparkConf): Int = {
> conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT)
> }
> Better retrieve effective UI port before probing.
>
> Cheers
>
> On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> So basically, to tell if the master is ready to accept slaves, just poll
>> http://master-node:4040 for an HTTP 200 response?
>>
>>
>> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
>> shivaram@eecs.berkeley.edu> wrote:
>>
>> > Yeah from what I remember it was set defensively. I don't know of a good
>> > way to check if the master is up though. I guess we could poll the
>> Master
>> > Web UI and see if we get a 200/ok response
>> >
>> > Shivaram
>> >
>> > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
>> > nicholas.chammas@gmail.com> wrote:
>> >
>> >> Check this out
>> >> <
>> >>
>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>> >> >
>> >> (from spark-ec2):
>> >>
>> >> # Start Master$BIN_FOLDER/start-master.sh
>> >
>> >
>> >> # Pause
>> >> sleep 20
>> >> # Start Workers$BIN_FOLDER/start-slaves.sh
>> >>
>> >> I know this was probably done defensively, but is there a more direct
>> way
>> >> to know when the master is ready?
>> >>
>> >> Nick
>> >>
>> >>
>> >
>>
>
>
Re: wait time between start master and start slaves
Posted by Ted Yu <yu...@gmail.com>.
>From SparkUI.scala :
def getUIPort(conf: SparkConf): Int = {
conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT)
}
Better retrieve effective UI port before probing.
Cheers
On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> So basically, to tell if the master is ready to accept slaves, just poll
> http://master-node:4040 for an HTTP 200 response?
>
>
> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
> > Yeah from what I remember it was set defensively. I don't know of a good
> > way to check if the master is up though. I guess we could poll the Master
> > Web UI and see if we get a 200/ok response
> >
> > Shivaram
> >
> > On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
> > nicholas.chammas@gmail.com> wrote:
> >
> >> Check this out
> >> <
> >>
> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
> >> >
> >> (from spark-ec2):
> >>
> >> # Start Master$BIN_FOLDER/start-master.sh
> >
> >
> >> # Pause
> >> sleep 20
> >> # Start Workers$BIN_FOLDER/start-slaves.sh
> >>
> >> I know this was probably done defensively, but is there a more direct
> way
> >> to know when the master is ready?
> >>
> >> Nick
> >>
> >>
> >
>
Re: wait time between start master and start slaves
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
Yeah thats the best I can think ok -- Not sure if there is a better way to
do it.
On Sat, Apr 11, 2015 at 2:38 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> So basically, to tell if the master is ready to accept slaves, just poll
> http://master-node:4040 for an HTTP 200 response?
>
>
> On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
> shivaram@eecs.berkeley.edu> wrote:
>
>> Yeah from what I remember it was set defensively. I don't know of a good
>> way to check if the master is up though. I guess we could poll the Master
>> Web UI and see if we get a 200/ok response
>>
>> Shivaram
>>
>> On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
>> nicholas.chammas@gmail.com> wrote:
>>
>>> Check this out
>>> <
>>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>>> >
>>> (from spark-ec2):
>>>
>>> # Start Master$BIN_FOLDER/start-master.sh
>>
>>
>>> # Pause
>>> sleep 20
>>> # Start Workers$BIN_FOLDER/start-slaves.sh
>>>
>>> I know this was probably done defensively, but is there a more direct way
>>> to know when the master is ready?
>>>
>>> Nick
>>>
>>>
>>
Re: wait time between start master and start slaves
Posted by Nicholas Chammas <ni...@gmail.com>.
So basically, to tell if the master is ready to accept slaves, just poll
http://master-node:4040 for an HTTP 200 response?
On Sat, Apr 11, 2015 at 2:42 PM Shivaram Venkataraman <
shivaram@eecs.berkeley.edu> wrote:
> Yeah from what I remember it was set defensively. I don't know of a good
> way to check if the master is up though. I guess we could poll the Master
> Web UI and see if we get a 200/ok response
>
> Shivaram
>
> On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> Check this out
>> <
>> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
>> >
>> (from spark-ec2):
>>
>> # Start Master$BIN_FOLDER/start-master.sh
>
>
>> # Pause
>> sleep 20
>> # Start Workers$BIN_FOLDER/start-slaves.sh
>>
>> I know this was probably done defensively, but is there a more direct way
>> to know when the master is ready?
>>
>> Nick
>>
>>
>
Re: wait time between start master and start slaves
Posted by Shivaram Venkataraman <sh...@eecs.berkeley.edu>.
Yeah from what I remember it was set defensively. I don't know of a good
way to check if the master is up though. I guess we could poll the Master
Web UI and see if we get a 200/ok response
Shivaram
On Fri, Apr 10, 2015 at 8:24 PM, Nicholas Chammas <
nicholas.chammas@gmail.com> wrote:
> Check this out
> <
> https://github.com/mesos/spark-ec2/blob/f0a48be1bb5aaeef508619a46065648beb8f1d92/spark-standalone/setup.sh#L26-L33
> >
> (from spark-ec2):
>
> # Start Master$BIN_FOLDER/start-master.sh
> # Pause
> sleep 20
> # Start Workers$BIN_FOLDER/start-slaves.sh
>
> I know this was probably done defensively, but is there a more direct way
> to know when the master is ready?
>
> Nick
>
>