You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Rodrick Brown <ro...@orchard-app.com> on 2016/04/15 04:05:29 UTC
decline_offer_duration for spark frameworks
I have hundreds of small spark jobs running on my Mesos cluster causing
starvation to other frameworks like Marathon on my cluster.
Is their a way to prevent these frameworks from getting offers so often?
Apr 15 02:00:12 prod-mesos-m-3.$SERVER.com mesos-master[10259]: I0415
02:00:12.503734 10266 master.cpp:3641] Processing DECLINE call for offers: [
50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-O7112667 ] for framework
50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-15936 (KafkaDirectConsumer[trades-topic])
at scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858.
\--
**Rodrick Brown** / Systems Engineer
+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
**Orchard Platform**
101 5th Avenue, 4th Floor, New York, NY 10003
[http://www.orchardplatform.com](http://www.orchardplatform.com/)
[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: decline_offer_duration for spark frameworks
Posted by Rodrick Brown <ro...@orchard-app.com>.
Hello Haosdent,
I'm running in Coarse gain mode the issue i'm seeing while looking at logs I
see tons of offers going to these frameworks which are started by chronos
using a simple python wrapper that calls my spark task. Once the spark jobs
are running I see offers being declined over and over while marathon is
starving for offers and I'm not able to restart my marathon process in a
timely manner.
I see about 1-4 offers an hour being given to spark.
I could not find any details on how set decline offer timeout in the spark
framework.
Any assistance would be helpful.
My cluster consists of 14 nodes each with 16 cores and 60GB of memory
I have about 120 or so spark tasks configured in Chronos which run very
fequently with about 10 long running tasks running on marathon.
\--
**Rodrick Brown** / Systems Engineer
+1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
**Orchard Platform**
101 5th Avenue, 4th Floor, New York, NY 10003
[http://www.orchardplatform.com](http://www.orchardplatform.com/)
[Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
On Apr 14 2016, at 11:09 pm, haosdent <haosdent@gmail.com> wrote:
> Hi, @Rodrick Which spark mode are you running? coarse-grained or fine-
grained?
>
>
>
> On Fri, Apr 15, 2016 at 10:05 AM, Rodrick Brown <[rodrick@orchard-
app.com](mailto:rodrick@orchard-app.com)> wrote:
>
>> I have hundreds of small spark jobs running on my Mesos cluster causing
starvation to other frameworks like Marathon on my cluster.
>>
>>
>>
>> Is their a way to prevent these frameworks from getting offers so often?
>>
>>
>>
>> Apr 15 02:00:12 prod-mesos-m-3.$SERVER.com mesos-master[10259]: I0415
02:00:12.503734 10266 master.cpp:3641] Processing DECLINE call for offers: [
50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-O7112667 ] for framework
50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-15936 (KafkaDirectConsumer[trades-topic])
at [scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858](http
://scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858).
>>
>>
>>
>>
>>
>>
>>
>> \--
>>
>> **Rodrick Brown** / Systems Engineer
>>
>> +1 917 445 6839 /
[rodrick@orchardplatform.com](mailto:charlie@orchardplatform.com)
>>
>> **Orchard Platform**
>>
>> 101 5th Avenue, 4th Floor, New York, NY 10003
>>
>> [http://www.orchardplatform.com](http://www.orchardplatform.com/)
>>
>> [Orchard Blog](http://www.orchardplatform.com/blog/) | [Marketplace Lending
Meetup](http://www.meetup.com/Peer-to-Peer-Lending-P2P/)
>>
>>
>>
>> **NOTICE TO RECIPIENTS**: This communication is confidential and intended
for the use of the addressee only. If you are not an intended recipient of
this communication, please delete it immediately and notify the sender by
return email. Unauthorized reading, dissemination, distribution or copying of
this communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase any
loan, security or any other financial product or instrument, nor is it an
offer to sell or a solicitation of an indication of interest to purchase any
products or services to any persons who are prohibited from receiving such
information under applicable law. The contents of this communication may not
be accurate or complete and are subject to change without notice. As such,
Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") makes
no representation regarding the accuracy or completeness of the information
contained herein. The intended recipient is advised to consult its own
professional advisors, including those specializing in legal, tax and
accounting matters. Orchard does not provide legal, tax or accounting advice.
>
>
>
>
>
> \--
>
> Best Regards,
>
> Haosdent Huang
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: decline_offer_duration for spark frameworks
Posted by haosdent <ha...@gmail.com>.
>I could not find any details on how set decline offer timeout in the spark
framework.
Seems spark framework don't support this while marathon support decline
offer duration.
Have you ever try set different role and reservation resources for
marathon? Related documents are [reservation.md](
https://github.com/apache/mesos/blob/master/docs/reservation.md) and [
weights.md](https://github.com/apache/mesos/blob/master/docs/weights.md)
On Fri, Apr 15, 2016 at 11:24 AM, David Greenberg <ds...@gmail.com>
wrote:
> Hi Rodrick,
> You should check out Cook (github.com/twosigma/cook). It acts as a server
> for Spark drivers, except that it can manage preemption and fair-sharing
> between those drivers, so that you don't encounter starvation. If you'd
> like to discuss it, I'd be happy to hop on a call.
>
> Best,
> David
>
> On Thu, Apr 14, 2016 at 11:09 PM haosdent <ha...@gmail.com> wrote:
>
>> Hi, @Rodrick Which spark mode are you running? coarse-grained
>> or fine-grained?
>>
>> On Fri, Apr 15, 2016 at 10:05 AM, Rodrick Brown <ro...@orchard-app.com>
>> wrote:
>>
>>> I have hundreds of small spark jobs running on my Mesos cluster
>>> causing starvation to other frameworks like Marathon on my cluster.
>>>
>>> Is their a way to prevent these frameworks from getting offers so often?
>>>
>>> Apr 15 02:00:12 prod-mesos-m-3.$SERVER.com mesos-master[10259]: I0415
>>> 02:00:12.503734 10266 master.cpp:3641] Processing DECLINE call for offers:
>>> [ 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-O7112667 ] for
>>> framework 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-15936 (KafkaDirectConsumer[trades-topic])
>>> at scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858.
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> *Rodrick Brown* / Systems Engineer
>>>
>>> +1 917 445 6839 / rodrick@orchardplatform.com
>>> <ch...@orchardplatform.com>
>>>
>>> *Orchard Platform*
>>>
>>> 101 5th Avenue, 4th Floor, New York, NY 10003
>>>
>>> http://www.orchardplatform.com
>>>
>>> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace
>>> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>>>
>>> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
>>> for the use of the addressee only. If you are not an intended recipient of
>>> this communication, please delete it immediately and notify the sender
>>> by return email. Unauthorized reading, dissemination, distribution or
>>> copying of this communication is prohibited. This communication does not constitute
>>> an offer to sell or a solicitation of an indication of interest to purchase
>>> any loan, security or any other financial product or instrument, nor is it
>>> an offer to sell or a solicitation of an indication of interest to purchase
>>> any products or services to any persons who are prohibited from receiving
>>> such information under applicable law. The contents of this communication
>>> may not be accurate or complete and are subject to change without notice.
>>> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
>>> "Orchard") makes no representation regarding the accuracy or
>>> completeness of the information contained herein. The intended recipient is
>>> advised to consult its own professional advisors, including those
>>> specializing in legal, tax and accounting matters. Orchard does not
>>> provide legal, tax or accounting advice.
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Haosdent Huang
>>
>
--
Best Regards,
Haosdent Huang
Re: decline_offer_duration for spark frameworks
Posted by David Greenberg <ds...@gmail.com>.
Hi Rodrick,
You should check out Cook (github.com/twosigma/cook). It acts as a server
for Spark drivers, except that it can manage preemption and fair-sharing
between those drivers, so that you don't encounter starvation. If you'd
like to discuss it, I'd be happy to hop on a call.
Best,
David
On Thu, Apr 14, 2016 at 11:09 PM haosdent <ha...@gmail.com> wrote:
> Hi, @Rodrick Which spark mode are you running? coarse-grained
> or fine-grained?
>
> On Fri, Apr 15, 2016 at 10:05 AM, Rodrick Brown <ro...@orchard-app.com>
> wrote:
>
>> I have hundreds of small spark jobs running on my Mesos cluster
>> causing starvation to other frameworks like Marathon on my cluster.
>>
>> Is their a way to prevent these frameworks from getting offers so often?
>>
>> Apr 15 02:00:12 prod-mesos-m-3.$SERVER.com mesos-master[10259]: I0415
>> 02:00:12.503734 10266 master.cpp:3641] Processing DECLINE call for offers:
>> [ 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-O7112667 ] for
>> framework 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-15936 (KafkaDirectConsumer[trades-topic])
>> at scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858.
>>
>>
>>
>>
>>
>> --
>>
>> *Rodrick Brown* / Systems Engineer
>>
>> +1 917 445 6839 / rodrick@orchardplatform.com
>> <ch...@orchardplatform.com>
>>
>> *Orchard Platform*
>>
>> 101 5th Avenue, 4th Floor, New York, NY 10003
>>
>> http://www.orchardplatform.com
>>
>> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace
>> Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>>
>> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
>> for the use of the addressee only. If you are not an intended recipient of
>> this communication, please delete it immediately and notify the sender
>> by return email. Unauthorized reading, dissemination, distribution or
>> copying of this communication is prohibited. This communication does not constitute
>> an offer to sell or a solicitation of an indication of interest to purchase
>> any loan, security or any other financial product or instrument, nor is it
>> an offer to sell or a solicitation of an indication of interest to purchase
>> any products or services to any persons who are prohibited from receiving
>> such information under applicable law. The contents of this communication
>> may not be accurate or complete and are subject to change without notice.
>> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
>> "Orchard") makes no representation regarding the accuracy or
>> completeness of the information contained herein. The intended recipient is
>> advised to consult its own professional advisors, including those
>> specializing in legal, tax and accounting matters. Orchard does not
>> provide legal, tax or accounting advice.
>>
>
>
>
> --
> Best Regards,
> Haosdent Huang
>
Re: decline_offer_duration for spark frameworks
Posted by haosdent <ha...@gmail.com>.
Hi, @Rodrick Which spark mode are you running? coarse-grained
or fine-grained?
On Fri, Apr 15, 2016 at 10:05 AM, Rodrick Brown <ro...@orchard-app.com>
wrote:
> I have hundreds of small spark jobs running on my Mesos cluster
> causing starvation to other frameworks like Marathon on my cluster.
>
> Is their a way to prevent these frameworks from getting offers so often?
>
> Apr 15 02:00:12 prod-mesos-m-3.$SERVER.com mesos-master[10259]: I0415
> 02:00:12.503734 10266 master.cpp:3641] Processing DECLINE call for offers:
> [ 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-O7112667 ] for
> framework 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-15936 (KafkaDirectConsumer[trades-topic])
> at scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858.
>
>
>
>
>
> --
>
> *Rodrick Brown* / Systems Engineer
>
> +1 917 445 6839 / rodrick@orchardplatform.com
> <ch...@orchardplatform.com>
>
> *Orchard Platform*
>
> 101 5th Avenue, 4th Floor, New York, NY 10003
>
> http://www.orchardplatform.com
>
> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending
> Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient of
> this communication, please delete it immediately and notify the sender by
> return email. Unauthorized reading, dissemination, distribution or copying
> of this communication is prohibited. This communication does not constitute
> an offer to sell or a solicitation of an indication of interest to purchase
> any loan, security or any other financial product or instrument, nor is it
> an offer to sell or a solicitation of an indication of interest to purchase
> any products or services to any persons who are prohibited from receiving
> such information under applicable law. The contents of this communication
> may not be accurate or complete and are subject to change without notice.
> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or completeness
> of the information contained herein. The intended recipient is advised to
> consult its own professional advisors, including those specializing in
> legal, tax and accounting matters. Orchard does not provide legal, tax or
> accounting advice.
>
--
Best Regards,
Haosdent Huang