You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by June Taylor <ju...@umn.edu> on 2016/04/13 16:32:24 UTC
Mesos Task History
I am noticing that recently our Completed Tasks and Terminated Frameworks
lists are empty. Where are these stored, and do they get automatically
cleared out at some interval?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
Re: Mesos master not joining cluster
Posted by shakeel <sh...@motortrak.com>.
Hi Shuai,
Please find an exert from the master in question logs below. It thinks
it is the leader.
The WebUI does not show any jobs and does not redirect to the current
master.
The other two masters are working properly, with the corect leader
showing the jobs and the other master redirecting.
I0414 12:51:44.156545 1068 detector.cpp:482] A new leading master
(UPID=master@192.168.128.191:5050) is detected
I0414 12:51:44.156906 1069 master.cpp:1606] The newly elected leader is
master@192.168.128.191:5050 with id 5a570a87-877c-497a-b91e-8726fce92959
Kind Regards
Shakeel Suffee
On 14/04/16 13:48, Shuai Lin wrote:
> Hi Shakeel, what do you mean by "one of the master was not participating
> in the quorum"? Can you paste related lines from the logs of that master?
>
> On Thu, Apr 14, 2016 at 8:44 PM, shakeel <shakeel.suffee@motortrak.com
> <ma...@motortrak.com>> wrote:
>
> Hi,
>
> I have three mesos master configured. They have all been working
> properly for a while and today I noticed one of the master was not
> participating in the quorum.
>
> A reboot did not resolve the problem.
>
> All three of the masters are configured with a quorum of 2.
>
> Has anyone experienced this problem before and what is the easy way of
> getting the master server back in the quorum?
>
> Thanks
>
>
> Kind Regards
> Shakeel Suffee
>
> --
> The information contained in this message is for the intended addressee
> only and may contain confidential and/or privileged information. If
> you are
> not the intended addressee, please delete this message and notify the
> sender; do not copy or distribute this message or disclose its
> contents to
> anyone. Any views or opinions expressed in this message are those of the
> author and do not necessarily represent those of Motortrak Limited or of
> any of its associated companies. No reliance may be placed on this
> message
> without written confirmation from an authorised representative of the
> company.
>
> Registered in England 3098391 V.A.T. Registered No. 667463890
>
>
--
The information contained in this message is for the intended addressee
only and may contain confidential and/or privileged information. If you are
not the intended addressee, please delete this message and notify the
sender; do not copy or distribute this message or disclose its contents to
anyone. Any views or opinions expressed in this message are those of the
author and do not necessarily represent those of Motortrak Limited or of
any of its associated companies. No reliance may be placed on this message
without written confirmation from an authorised representative of the
company.
Registered in England 3098391 V.A.T. Registered No. 667463890
Re: Mesos master not joining cluster
Posted by Shuai Lin <li...@gmail.com>.
Hi Shakeel, what do you mean by "one of the master was not participating in
the quorum"? Can you paste related lines from the logs of that master?
On Thu, Apr 14, 2016 at 8:44 PM, shakeel <sh...@motortrak.com>
wrote:
> Hi,
>
> I have three mesos master configured. They have all been working
> properly for a while and today I noticed one of the master was not
> participating in the quorum.
>
> A reboot did not resolve the problem.
>
> All three of the masters are configured with a quorum of 2.
>
> Has anyone experienced this problem before and what is the easy way of
> getting the master server back in the quorum?
>
> Thanks
>
>
> Kind Regards
> Shakeel Suffee
>
> --
> The information contained in this message is for the intended addressee
> only and may contain confidential and/or privileged information. If you are
> not the intended addressee, please delete this message and notify the
> sender; do not copy or distribute this message or disclose its contents to
> anyone. Any views or opinions expressed in this message are those of the
> author and do not necessarily represent those of Motortrak Limited or of
> any of its associated companies. No reliance may be placed on this message
> without written confirmation from an authorised representative of the
> company.
>
> Registered in England 3098391 V.A.T. Registered No. 667463890
>
Mesos master not joining cluster
Posted by shakeel <sh...@motortrak.com>.
Hi,
I have three mesos master configured. They have all been working
properly for a while and today I noticed one of the master was not
participating in the quorum.
A reboot did not resolve the problem.
All three of the masters are configured with a quorum of 2.
Has anyone experienced this problem before and what is the easy way of
getting the master server back in the quorum?
Thanks
Kind Regards
Shakeel Suffee
--
The information contained in this message is for the intended addressee
only and may contain confidential and/or privileged information. If you are
not the intended addressee, please delete this message and notify the
sender; do not copy or distribute this message or disclose its contents to
anyone. Any views or opinions expressed in this message are those of the
author and do not necessarily represent those of Motortrak Limited or of
any of its associated companies. No reliance may be placed on this message
without written confirmation from an authorised representative of the
company.
Registered in England 3098391 V.A.T. Registered No. 667463890
Re: Mesos Task History
Posted by Adam Bordelon <ad...@mesosphere.io>.
You might also want to check out TwoSigma's Satellite:
https://github.com/twosigma/satellite
Sunil's Satellite talk from MesosCon:
https://www.youtube.com/watch?v=yLkc17HFEb8
On Thu, Apr 14, 2016 at 9:16 AM, Dick Davies <di...@hellooperator.net> wrote:
> We just grab them with collectds mesos plugin and log to Graphite,
> gives us long term trend details.
>
> https://github.com/rayrod2030/collectd-mesos
>
> Haven't used this one but it supposedly does per-task metric collection:
>
> https://github.com/bobrik/collectd-mesos-tasks
>
> On 14 April 2016 at 13:37, June Taylor <ju...@umn.edu> wrote:
> > Adam,
> >
> > Is there a way to keep this history?
> >
> >
> > Thanks,
> > June Taylor
> > System Administrator, Minnesota Population Center
> > University of Minnesota
> >
> > On Wed, Apr 13, 2016 at 4:32 PM, Adam Bordelon <ad...@mesosphere.io>
> wrote:
> >>
> >> Yes, these counters are only kept in-memory, so any time a Mesos master
> >> starts, its counters are initialized to 0.
> >>
> >> On Wed, Apr 13, 2016 at 9:38 AM, June Taylor <ju...@umn.edu> wrote:
> >>>
> >>> We have a single master at the moment. Does the task history get
> cleared
> >>> when the mesos-master restarts?
> >>>
> >>>
> >>> Thanks,
> >>> June Taylor
> >>> System Administrator, Minnesota Population Center
> >>> University of Minnesota
> >>>
> >>> On Wed, Apr 13, 2016 at 11:33 AM, Pradeep Chhetri
> >>> <pr...@gmail.com> wrote:
> >>>>
> >>>> Yes, they get cleaned up whenever the mesos master leader failover
> >>>> happens.
> >>>>
> >>>> On Wed, Apr 13, 2016 at 3:32 PM, June Taylor <ju...@umn.edu> wrote:
> >>>>>
> >>>>> I am noticing that recently our Completed Tasks and Terminated
> >>>>> Frameworks lists are empty. Where are these stored, and do they get
> >>>>> automatically cleared out at some interval?
> >>>>>
> >>>>> Thanks,
> >>>>> June Taylor
> >>>>> System Administrator, Minnesota Population Center
> >>>>> University of Minnesota
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Pradeep Chhetri
> >>>
> >>>
> >>
> >
>
Re: Mesos Task History
Posted by Dick Davies <di...@hellooperator.net>.
We just grab them with collectds mesos plugin and log to Graphite,
gives us long term trend details.
https://github.com/rayrod2030/collectd-mesos
Haven't used this one but it supposedly does per-task metric collection:
https://github.com/bobrik/collectd-mesos-tasks
On 14 April 2016 at 13:37, June Taylor <ju...@umn.edu> wrote:
> Adam,
>
> Is there a way to keep this history?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Apr 13, 2016 at 4:32 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
>>
>> Yes, these counters are only kept in-memory, so any time a Mesos master
>> starts, its counters are initialized to 0.
>>
>> On Wed, Apr 13, 2016 at 9:38 AM, June Taylor <ju...@umn.edu> wrote:
>>>
>>> We have a single master at the moment. Does the task history get cleared
>>> when the mesos-master restarts?
>>>
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>> On Wed, Apr 13, 2016 at 11:33 AM, Pradeep Chhetri
>>> <pr...@gmail.com> wrote:
>>>>
>>>> Yes, they get cleaned up whenever the mesos master leader failover
>>>> happens.
>>>>
>>>> On Wed, Apr 13, 2016 at 3:32 PM, June Taylor <ju...@umn.edu> wrote:
>>>>>
>>>>> I am noticing that recently our Completed Tasks and Terminated
>>>>> Frameworks lists are empty. Where are these stored, and do they get
>>>>> automatically cleared out at some interval?
>>>>>
>>>>> Thanks,
>>>>> June Taylor
>>>>> System Administrator, Minnesota Population Center
>>>>> University of Minnesota
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Pradeep Chhetri
>>>
>>>
>>
>
Re: Mesos Task History
Posted by June Taylor <ju...@umn.edu>.
Adam,
Is there a way to keep this history?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Wed, Apr 13, 2016 at 4:32 PM, Adam Bordelon <ad...@mesosphere.io> wrote:
> Yes, these counters are only kept in-memory, so any time a Mesos master
> starts, its counters are initialized to 0.
>
> On Wed, Apr 13, 2016 at 9:38 AM, June Taylor <ju...@umn.edu> wrote:
>
>> We have a single master at the moment. Does the task history get cleared
>> when the mesos-master restarts?
>>
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>> On Wed, Apr 13, 2016 at 11:33 AM, Pradeep Chhetri <
>> pradeep.chhetri89@gmail.com> wrote:
>>
>>> Yes, they get cleaned up whenever the mesos master leader failover
>>> happens.
>>>
>>> On Wed, Apr 13, 2016 at 3:32 PM, June Taylor <ju...@umn.edu> wrote:
>>>
>>>> I am noticing that recently our Completed Tasks and Terminated
>>>> Frameworks lists are empty. Where are these stored, and do they get
>>>> automatically cleared out at some interval?
>>>>
>>>> Thanks,
>>>> June Taylor
>>>> System Administrator, Minnesota Population Center
>>>> University of Minnesota
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Pradeep Chhetri
>>>
>>
>>
>
Re: Mesos Task History
Posted by Adam Bordelon <ad...@mesosphere.io>.
Yes, these counters are only kept in-memory, so any time a Mesos master
starts, its counters are initialized to 0.
On Wed, Apr 13, 2016 at 9:38 AM, June Taylor <ju...@umn.edu> wrote:
> We have a single master at the moment. Does the task history get cleared
> when the mesos-master restarts?
>
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
> On Wed, Apr 13, 2016 at 11:33 AM, Pradeep Chhetri <
> pradeep.chhetri89@gmail.com> wrote:
>
>> Yes, they get cleaned up whenever the mesos master leader failover
>> happens.
>>
>> On Wed, Apr 13, 2016 at 3:32 PM, June Taylor <ju...@umn.edu> wrote:
>>
>>> I am noticing that recently our Completed Tasks and Terminated
>>> Frameworks lists are empty. Where are these stored, and do they get
>>> automatically cleared out at some interval?
>>>
>>> Thanks,
>>> June Taylor
>>> System Administrator, Minnesota Population Center
>>> University of Minnesota
>>>
>>
>>
>>
>> --
>> Regards,
>> Pradeep Chhetri
>>
>
>
Re: Mesos Task History
Posted by June Taylor <ju...@umn.edu>.
We have a single master at the moment. Does the task history get cleared
when the mesos-master restarts?
Thanks,
June Taylor
System Administrator, Minnesota Population Center
University of Minnesota
On Wed, Apr 13, 2016 at 11:33 AM, Pradeep Chhetri <
pradeep.chhetri89@gmail.com> wrote:
> Yes, they get cleaned up whenever the mesos master leader failover happens.
>
> On Wed, Apr 13, 2016 at 3:32 PM, June Taylor <ju...@umn.edu> wrote:
>
>> I am noticing that recently our Completed Tasks and Terminated Frameworks
>> lists are empty. Where are these stored, and do they get automatically
>> cleared out at some interval?
>>
>> Thanks,
>> June Taylor
>> System Administrator, Minnesota Population Center
>> University of Minnesota
>>
>
>
>
> --
> Regards,
> Pradeep Chhetri
>
Re: Mesos Task History
Posted by Pradeep Chhetri <pr...@gmail.com>.
Yes, they get cleaned up whenever the mesos master leader failover happens.
On Wed, Apr 13, 2016 at 3:32 PM, June Taylor <ju...@umn.edu> wrote:
> I am noticing that recently our Completed Tasks and Terminated Frameworks
> lists are empty. Where are these stored, and do they get automatically
> cleared out at some interval?
>
> Thanks,
> June Taylor
> System Administrator, Minnesota Population Center
> University of Minnesota
>
--
Regards,
Pradeep Chhetri