You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Puneet Kumar (Jira)" <ji...@apache.org> on 2020/08/15 08:25:00 UTC

[jira] [Commented] (MESOS-10143) Outstanding Offers accumulating

    [ https://issues.apache.org/jira/browse/MESOS-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17178186#comment-17178186 ] 

Puneet Kumar commented on MESOS-10143:
--------------------------------------

We reduced average latency of Scheduler::resourceOffers from ~50 ms to ~15ms by eliminating non-essential workload from our java application that implements Scheduler callbacks. After that, we haven't observed this issue till date. Our application is handling ~400 offers per second at average latency of 15 ms for Scheduler::resourceOffers.

[~bmahler] Thanks for describing how to obtain metrics for scheduler library.

 

> Outstanding Offers accumulating
> -------------------------------
>
>                 Key: MESOS-10143
>                 URL: https://issues.apache.org/jira/browse/MESOS-10143
>             Project: Mesos
>          Issue Type: Bug
>          Components: master, scheduler driver
>    Affects Versions: 1.7.0
>         Environment: Mesos Version 1.7.0
> JDK 8.0
>            Reporter: Puneet Kumar
>            Priority: Minor
>
> We manage an Apache Mesos cluster version 1.7.0. We have written a framework in Java that schedules tasks to Mesos master at a rate of 300 TPS. Everything works fine for almost 24 hours but then outstanding offers accumulate & saturate within 15 minutes. Outstanding offers aren't reclaimed by Mesos master. We observe "RescindOffer" messages in verbose (GLOG v=3) framework logs but outstanding offers don't reduce. New resources aren't offered to framework when outstanding offers saturate. We have to restart the scheduler to reset outstanding offers to zero.
> Any suggestions to debug this issue are welcome.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)