You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Badal Naik <th...@icloud.com> on 2015/10/09 14:16:48 UTC

After restarting cluster task disappeared

Hello Mesos-Users,

I have set up 3 node mess cluster with ubuntu 14.04. i have started zookeeper,Mesos and marathon. Every thing working fine expect three things.

1) When i restart the whole cluster mesos does not show completed tasks. is it expected behaviour? if not what i should do?

2) in mesos web ui i’m not able to see staged/started/finished/killed/failed/lost task numbers even when tasks are running.

3) Every zookeeper instance throws this exception regularly:

 2015-10-09 17:27:26,302 [myid:3] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)




Here is my Mesos-master configuration:

mesos master --ip=10.1.0.72  --work_dir=/var/lib/mesos-master --zk=file:///etc/mesos/conf/zk --quorum=file:///etc/mesos/conf/quorum
	
	Where zk=zk://zoo.service.consul:2181/mesos
              quorum=2
              


Mesos-Slave Configuration:

mesos slave --work_dir=/var/lib/mesos-slave --ip=10.1.0.72 --hostname=10.1.0.72 --strict=false  --master=file:///etc/mesos/conf/master FrameworkInfo.checkpoint=True
      


Marathon Configuration:

java -jar /opt/marathon.jar  --master zk://zoo.service.consul:2181/mesos  --zk zk://zoo.service.consul:2181/marathon  --ha --hostname 10.1.0.72  --checkpoint




Zookeeper configs with java version "1.8.0_45":



dataDir=/var/lib/zookeeper
clientPort=2181
tickTime=2000
initLimit=10
syncLimit=20


autopurge.purgeInterval=0


zookeeper.connection.timeout.ms=6000
server.1=10.1.0.70:2888:3888
server.2=10.1.0.71:2888:3888
server.3=10.1.0.72:2888:3888

And different myid has been given.


Can Anyone Help!!!



RE: After restarting cluster task disappeared

Posted by Badal Naik <th...@icloud.com>.
Got it. I have build mesos from source so I need to initialize log-dir manually for once.
And now things are working.Thank you 

-----Original Message-----
From: "Badal Naik" <th...@icloud.com>
Sent: ‎09/‎10/‎2015 06:38 PM
To: "user@mesos.apache.org" <us...@mesos.apache.org>
Subject: Re: After restarting cluster task disappeared

No i have not deleted anything.
i  am just restarting physical nodes and then i am not able to check old completed tasks.And after restarting many times marathon and mess just stay out of sync and when i restart marathon service both get synced.




On 09-Oct-2015, at 6:32 pm, haosdent <ha...@gmail.com> wrote:


For #1, do you delete something in your work_dir or zookeeper?
For #3, is this zookeeper issue related to yours http://stackoverflow.com/questions/15842553/zookeeper-network-ensemble-does-not-start-appropiately ?


On Fri, Oct 9, 2015 at 8:30 PM, craig w <co...@gmail.com> wrote:

I'm not sure about #3. I have seen things go awry when restarting the whole cluster. When doing an upgrade from mesos 0.23.0 to 0.24.1, I restarted all of the mesos-masters. Waited a few moments for a leader to be elected, then restarted the slaves. When I went back to look at Marathon all of the tasks were being redeployed, as though they had all been killed off for some reason. That wasn't what I expected to happen since the upgrade was suppose to be as simple as install and restart. Perhaps you're experiencing a similar issue?



On Fri, Oct 9, 2015 at 8:25 AM, Badal Naik <th...@icloud.com> wrote:

Any idea about #1 ?


Any one has experienced #3 ?


On 09-Oct-2015, at 5:53 pm, craig w <co...@gmail.com> wrote:


With regards to item #2, I saw the same issue. it's been fixed in mesos 0.25 (release candidates are out now), see https://issues.apache.org/jira/browse/MESOS-3282.



On Fri, Oct 9, 2015 at 8:16 AM, Badal Naik <th...@icloud.com> wrote:

Hello Mesos-Users,


I have set up 3 node mess cluster with ubuntu 14.04. i have started zookeeper,Mesos and marathon. Every thing working fine expect three things.


1) When i restart the whole cluster mesos does not show completed tasks. is it expected behaviour? if not what i should do?


2) in mesos web ui i’m not able to see staged/started/finished/killed/failed/lost task numbers even when tasks are running.


3) Every zookeeper instance throws this exception regularly:


 2015-10-09 17:27:26,302 [myid:3] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)








Here is my Mesos-master configuration:


mesos master --ip=10.1.0.72  --work_dir=/var/lib/mesos-master --zk=file:///etc/mesos/conf/zk --quorum=file:///etc/mesos/conf/quorum
Where zk=zk://zoo.service.consul:2181/mesos
              quorum=2
              




Mesos-Slave Configuration:


mesos slave --work_dir=/var/lib/mesos-slave --ip=10.1.0.72 --hostname=10.1.0.72 --strict=false  --master=file:///etc/mesos/conf/master FrameworkInfo.checkpoint=True
      




Marathon Configuration:


java -jar /opt/marathon.jar  --master zk://zoo.service.consul:2181/mesos  --zk zk://zoo.service.consul:2181/marathon  --ha --hostname 10.1.0.72  --checkpoint








Zookeeper configs with java version "1.8.0_45":






dataDir=/var/lib/zookeeper
clientPort=2181
tickTime=2000
initLimit=10
syncLimit=20



autopurge.purgeInterval=0



zookeeper.connection.timeout.ms=6000
server.1=10.1.0.70:2888:3888
server.2=10.1.0.71:2888:3888
server.3=10.1.0.72:2888:3888


And different myid has been given.




Can Anyone Help!!!







-- 

https://github.com/mindscratchhttps://www.google.com/+CraigWickesserhttps://twitter.com/mind_scratchhttps://twitter.com/craig_links




-- 

https://github.com/mindscratchhttps://www.google.com/+CraigWickesserhttps://twitter.com/mind_scratchhttps://twitter.com/craig_links




-- 

Best Regards,

Haosdent Huang

Re: After restarting cluster task disappeared

Posted by Badal Naik <th...@icloud.com>.
No i have not deleted anything.
i  am just restarting physical nodes and then i am not able to check old completed tasks.And after restarting many times marathon and mess just stay out of sync and when i restart marathon service both get synced.


> On 09-Oct-2015, at 6:32 pm, haosdent <ha...@gmail.com> wrote:
> 
> For #1, do you delete something in your work_dir or zookeeper?
> For #3, is this zookeeper issue related to yours http://stackoverflow.com/questions/15842553/zookeeper-network-ensemble-does-not-start-appropiately <http://stackoverflow.com/questions/15842553/zookeeper-network-ensemble-does-not-start-appropiately> ?
> 
> On Fri, Oct 9, 2015 at 8:30 PM, craig w <codecraig@gmail.com <ma...@gmail.com>> wrote:
> I'm not sure about #3. I have seen things go awry when restarting the whole cluster. When doing an upgrade from mesos 0.23.0 to 0.24.1, I restarted all of the mesos-masters. Waited a few moments for a leader to be elected, then restarted the slaves. When I went back to look at Marathon all of the tasks were being redeployed, as though they had all been killed off for some reason. That wasn't what I expected to happen since the upgrade was suppose to be as simple as install and restart. Perhaps you're experiencing a similar issue?
> 
> On Fri, Oct 9, 2015 at 8:25 AM, Badal Naik <think_high@icloud.com <ma...@icloud.com>> wrote:
> Any idea about #1 ?
> 
> Any one has experienced #3 ?
> 
>> On 09-Oct-2015, at 5:53 pm, craig w <codecraig@gmail.com <ma...@gmail.com>> wrote:
>> 
>> With regards to item #2, I saw the same issue. it's been fixed in mesos 0.25 (release candidates are out now), see https://issues.apache.org/jira/browse/MESOS-3282 <https://issues.apache.org/jira/browse/MESOS-3282>.
>> 
>> On Fri, Oct 9, 2015 at 8:16 AM, Badal Naik <think_high@icloud.com <ma...@icloud.com>> wrote:
>> Hello Mesos-Users,
>> 
>> I have set up 3 node mess cluster with ubuntu 14.04. i have started zookeeper,Mesos and marathon. Every thing working fine expect three things.
>> 
>> 1) When i restart the whole cluster mesos does not show completed tasks. is it expected behaviour? if not what i should do?
>> 
>> 2) in mesos web ui i’m not able to see staged/started/finished/killed/failed/lost task numbers even when tasks are running.
>> 
>> 3) Every zookeeper instance throws this exception regularly:
>> 
>>  2015-10-09 17:27:26,302 [myid:3] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue
>> java.lang.InterruptedException
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
>> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
>> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
>> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
>> 
>> 
>> 
>> 
>> Here is my Mesos-master configuration:
>> 
>> mesos master --ip=10.1.0.72  --work_dir=/var/lib/mesos-master --zk=file:///etc/mesos/conf/zk <> --quorum=file:///etc/mesos/conf/quorum <>
>> 	
>> 	Where zk=zk://zoo.service.consul:2181/mesos <>
>>               quorum=2
>>               
>> 
>> 
>> Mesos-Slave Configuration:
>> 
>> mesos slave --work_dir=/var/lib/mesos-slave --ip=10.1.0.72 --hostname=10.1.0.72 --strict=false  --master=file:///etc/mesos/conf/master <> FrameworkInfo.checkpoint=True
>>       
>> 
>> 
>> Marathon Configuration:
>> 
>> java -jar /opt/marathon.jar  --master zk://zoo.service.consul:2181/mesos <>  --zk zk://zoo.service.consul:2181/marathon <>  --ha --hostname 10.1.0.72  --checkpoint
>> 
>> 
>> 
>> 
>> Zookeeper configs with java version "1.8.0_45":
>> 
>> 
>> 
>> dataDir=/var/lib/zookeeper
>> clientPort=2181
>> tickTime=2000
>> initLimit=10
>> syncLimit=20
>> 
>> 
>> autopurge.purgeInterval=0
>> 
>> 
>> zookeeper.connection.timeout.ms <http://zookeeper.connection.timeout.ms/>=6000
>> server.1=10.1.0.70:2888:3888
>> server.2=10.1.0.71:2888:3888
>> server.3=10.1.0.72:2888:3888
>> 
>> And different myid has been given.
>> 
>> 
>> Can Anyone Help!!!
>> 
>> 
>> 
>> 
>> 
>> -- 
>> https://github.com/mindscratch <https://github.com/mindscratch>
>> https://www.google.com/+CraigWickesser <https://www.google.com/+CraigWickesser>
>> https://twitter.com/mind_scratch <https://twitter.com/mind_scratch>
>> https://twitter.com/craig_links <https://twitter.com/craig_links>
> 
> 
> 
> 
> -- 
> https://github.com/mindscratch <https://github.com/mindscratch>
> https://www.google.com/+CraigWickesser <https://www.google.com/+CraigWickesser>
> https://twitter.com/mind_scratch <https://twitter.com/mind_scratch>
> https://twitter.com/craig_links <https://twitter.com/craig_links>
> 
> 
> 
> -- 
> Best Regards,
> Haosdent Huang


Re: After restarting cluster task disappeared

Posted by haosdent <ha...@gmail.com>.
For #1, do you delete something in your work_dir or zookeeper?
For #3, is this zookeeper issue related to yours
http://stackoverflow.com/questions/15842553/zookeeper-network-ensemble-does-not-start-appropiately
?

On Fri, Oct 9, 2015 at 8:30 PM, craig w <co...@gmail.com> wrote:

> I'm not sure about #3. I have seen things go awry when restarting the
> whole cluster. When doing an upgrade from mesos 0.23.0 to 0.24.1, I
> restarted all of the mesos-masters. Waited a few moments for a leader to be
> elected, then restarted the slaves. When I went back to look at Marathon
> all of the tasks were being redeployed, as though they had all been killed
> off for some reason. That wasn't what I expected to happen since the
> upgrade was suppose to be as simple as install and restart. Perhaps you're
> experiencing a similar issue?
>
> On Fri, Oct 9, 2015 at 8:25 AM, Badal Naik <th...@icloud.com> wrote:
>
>> Any idea about #1 ?
>>
>> Any one has experienced #3 ?
>>
>> On 09-Oct-2015, at 5:53 pm, craig w <co...@gmail.com> wrote:
>>
>> With regards to item #2, I saw the same issue. it's been fixed in mesos
>> 0.25 (release candidates are out now), see
>> https://issues.apache.org/jira/browse/MESOS-3282.
>>
>> On Fri, Oct 9, 2015 at 8:16 AM, Badal Naik <th...@icloud.com> wrote:
>>
>>> Hello Mesos-Users,
>>>
>>> I have set up 3 node mess cluster with ubuntu 14.04. i have started
>>> zookeeper,Mesos and marathon. Every thing working fine expect three things.
>>>
>>> 1) When i restart the whole cluster mesos does not show completed tasks.
>>> is it expected behaviour? if not what i should do?
>>>
>>> 2) in mesos web ui i’m not able to see
>>> staged/started/finished/killed/failed/lost task numbers even when tasks are
>>> running.
>>>
>>> 3) Every zookeeper instance throws this exception regularly:
>>>
>>>  2015-10-09 17:27:26,302 [myid:3] - WARN
>>> [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while
>>> waiting for message on queue
>>> java.lang.InterruptedException
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>>> at
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>>> at
>>> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
>>> at
>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
>>> at
>>> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
>>> at
>>> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
>>>
>>>
>>>
>>>
>>> *Here is my Mesos-master configuration:*
>>>
>>> mesos master --ip=10.1.0.72  --work_dir=/var/lib/mesos-master --zk=
>>> file:///etc/mesos/conf/zk --quorum=file:///etc/mesos/conf/quorum
>>> Where zk=zk://zoo.service.consul:2181/mesos
>>>               quorum=2
>>>
>>>
>>>
>>> *Mesos-Slave Configuration:*
>>>
>>> mesos slave --work_dir=/var/lib/mesos-slave --ip=10.1.0.72
>>> --hostname=10.1.0.72 --strict=false  --master=
>>> file:///etc/mesos/conf/master FrameworkInfo.checkpoint=True
>>>
>>>
>>>
>>> *Marathon Configuration:*
>>>
>>> java -jar /opt/marathon.jar  --master zk://zoo.service.consul:2181/mesos
>>> --zk zk://zoo.service.consul:2181/marathon  --ha --hostname 10.1.0.72
>>> --checkpoint
>>>
>>>
>>>
>>>
>>> *Zookeeper configs with java version *"1.8.0_45"*:*
>>>
>>>
>>>
>>> dataDir=/var/lib/zookeeper
>>> clientPort=2181
>>> tickTime=2000
>>> initLimit=10
>>> syncLimit=20
>>>
>>>
>>> autopurge.purgeInterval=0
>>>
>>>
>>> zookeeper.connection.timeout.ms=6000
>>> server.1=10.1.0.70:2888:3888
>>> server.2=10.1.0.71:2888:3888
>>> server.3=10.1.0.72:2888:3888
>>>
>>> And different *myid* has been given.
>>>
>>>
>>> Can Anyone Help!!!
>>>
>>>
>>>
>>
>>
>> --
>>
>> https://github.com/mindscratch
>> https://www.google.com/+CraigWickesser
>> https://twitter.com/mind_scratch
>> https://twitter.com/craig_links
>>
>>
>>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
>


-- 
Best Regards,
Haosdent Huang

Re: After restarting cluster task disappeared

Posted by craig w <co...@gmail.com>.
I'm not sure about #3. I have seen things go awry when restarting the whole
cluster. When doing an upgrade from mesos 0.23.0 to 0.24.1, I restarted all
of the mesos-masters. Waited a few moments for a leader to be elected, then
restarted the slaves. When I went back to look at Marathon all of the tasks
were being redeployed, as though they had all been killed off for some
reason. That wasn't what I expected to happen since the upgrade was suppose
to be as simple as install and restart. Perhaps you're experiencing a
similar issue?

On Fri, Oct 9, 2015 at 8:25 AM, Badal Naik <th...@icloud.com> wrote:

> Any idea about #1 ?
>
> Any one has experienced #3 ?
>
> On 09-Oct-2015, at 5:53 pm, craig w <co...@gmail.com> wrote:
>
> With regards to item #2, I saw the same issue. it's been fixed in mesos
> 0.25 (release candidates are out now), see
> https://issues.apache.org/jira/browse/MESOS-3282.
>
> On Fri, Oct 9, 2015 at 8:16 AM, Badal Naik <th...@icloud.com> wrote:
>
>> Hello Mesos-Users,
>>
>> I have set up 3 node mess cluster with ubuntu 14.04. i have started
>> zookeeper,Mesos and marathon. Every thing working fine expect three things.
>>
>> 1) When i restart the whole cluster mesos does not show completed tasks.
>> is it expected behaviour? if not what i should do?
>>
>> 2) in mesos web ui i’m not able to see
>> staged/started/finished/killed/failed/lost task numbers even when tasks are
>> running.
>>
>> 3) Every zookeeper instance throws this exception regularly:
>>
>>  2015-10-09 17:27:26,302 [myid:3] - WARN
>> [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while
>> waiting for message on queue
>> java.lang.InterruptedException
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>> at
>> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
>> at
>> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
>>
>>
>>
>>
>> *Here is my Mesos-master configuration:*
>>
>> mesos master --ip=10.1.0.72  --work_dir=/var/lib/mesos-master --zk=
>> file:///etc/mesos/conf/zk --quorum=file:///etc/mesos/conf/quorum
>> Where zk=zk://zoo.service.consul:2181/mesos
>>               quorum=2
>>
>>
>>
>> *Mesos-Slave Configuration:*
>>
>> mesos slave --work_dir=/var/lib/mesos-slave --ip=10.1.0.72
>> --hostname=10.1.0.72 --strict=false  --master=
>> file:///etc/mesos/conf/master FrameworkInfo.checkpoint=True
>>
>>
>>
>> *Marathon Configuration:*
>>
>> java -jar /opt/marathon.jar  --master zk://zoo.service.consul:2181/mesos
>> --zk zk://zoo.service.consul:2181/marathon  --ha --hostname 10.1.0.72
>> --checkpoint
>>
>>
>>
>>
>> *Zookeeper configs with java version *"1.8.0_45"*:*
>>
>>
>>
>> dataDir=/var/lib/zookeeper
>> clientPort=2181
>> tickTime=2000
>> initLimit=10
>> syncLimit=20
>>
>>
>> autopurge.purgeInterval=0
>>
>>
>> zookeeper.connection.timeout.ms=6000
>> server.1=10.1.0.70:2888:3888
>> server.2=10.1.0.71:2888:3888
>> server.3=10.1.0.72:2888:3888
>>
>> And different *myid* has been given.
>>
>>
>> Can Anyone Help!!!
>>
>>
>>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
>
>
>


-- 

https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links

Re: After restarting cluster task disappeared

Posted by Badal Naik <th...@icloud.com>.
Any idea about #1 ?

Any one has experienced #3 ?
> On 09-Oct-2015, at 5:53 pm, craig w <co...@gmail.com> wrote:
> 
> With regards to item #2, I saw the same issue. it's been fixed in mesos 0.25 (release candidates are out now), see https://issues.apache.org/jira/browse/MESOS-3282 <https://issues.apache.org/jira/browse/MESOS-3282>.
> 
> On Fri, Oct 9, 2015 at 8:16 AM, Badal Naik <think_high@icloud.com <ma...@icloud.com>> wrote:
> Hello Mesos-Users,
> 
> I have set up 3 node mess cluster with ubuntu 14.04. i have started zookeeper,Mesos and marathon. Every thing working fine expect three things.
> 
> 1) When i restart the whole cluster mesos does not show completed tasks. is it expected behaviour? if not what i should do?
> 
> 2) in mesos web ui i’m not able to see staged/started/finished/killed/failed/lost task numbers even when tasks are running.
> 
> 3) Every zookeeper instance throws this exception regularly:
> 
>  2015-10-09 17:27:26,302 [myid:3] - WARN  [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while waiting for message on queue
> java.lang.InterruptedException
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> 	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
> 	at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
> 
> 
> 
> 
> Here is my Mesos-master configuration:
> 
> mesos master --ip=10.1.0.72  --work_dir=/var/lib/mesos-master --zk=file:///etc/mesos/conf/zk <> --quorum=file:///etc/mesos/conf/quorum <>
> 	
> 	Where zk=zk://zoo.service.consul:2181/mesos <>
>               quorum=2
>               
> 
> 
> Mesos-Slave Configuration:
> 
> mesos slave --work_dir=/var/lib/mesos-slave --ip=10.1.0.72 --hostname=10.1.0.72 --strict=false  --master=file:///etc/mesos/conf/master <> FrameworkInfo.checkpoint=True
>       
> 
> 
> Marathon Configuration:
> 
> java -jar /opt/marathon.jar  --master zk://zoo.service.consul:2181/mesos <>  --zk zk://zoo.service.consul:2181/marathon <>  --ha --hostname 10.1.0.72  --checkpoint
> 
> 
> 
> 
> Zookeeper configs with java version "1.8.0_45":
> 
> 
> 
> dataDir=/var/lib/zookeeper
> clientPort=2181
> tickTime=2000
> initLimit=10
> syncLimit=20
> 
> 
> autopurge.purgeInterval=0
> 
> 
> zookeeper.connection.timeout.ms <http://zookeeper.connection.timeout.ms/>=6000
> server.1=10.1.0.70:2888:3888
> server.2=10.1.0.71:2888:3888
> server.3=10.1.0.72:2888:3888
> 
> And different myid has been given.
> 
> 
> Can Anyone Help!!!
> 
> 
> 
> 
> 
> -- 
> https://github.com/mindscratch <https://github.com/mindscratch>
> https://www.google.com/+CraigWickesser <https://www.google.com/+CraigWickesser>
> https://twitter.com/mind_scratch <https://twitter.com/mind_scratch>
> https://twitter.com/craig_links <https://twitter.com/craig_links>


Re: After restarting cluster task disappeared

Posted by craig w <co...@gmail.com>.
With regards to item #2, I saw the same issue. it's been fixed in mesos
0.25 (release candidates are out now), see
https://issues.apache.org/jira/browse/MESOS-3282.

On Fri, Oct 9, 2015 at 8:16 AM, Badal Naik <th...@icloud.com> wrote:

> Hello Mesos-Users,
>
> I have set up 3 node mess cluster with ubuntu 14.04. i have started
> zookeeper,Mesos and marathon. Every thing working fine expect three things.
>
> 1) When i restart the whole cluster mesos does not show completed tasks.
> is it expected behaviour? if not what i should do?
>
> 2) in mesos web ui i’m not able to see
> staged/started/finished/killed/failed/lost task numbers even when tasks are
> running.
>
> 3) Every zookeeper instance throws this exception regularly:
>
>  2015-10-09 17:27:26,302 [myid:3] - WARN
> [SendWorker:1:QuorumCnxManager$SendWorker@679] - Interrupted while
> waiting for message on queue
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> at
> java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
> at
> org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
>
>
>
>
> *Here is my Mesos-master configuration:*
>
> mesos master --ip=10.1.0.72  --work_dir=/var/lib/mesos-master --zk=
> file:///etc/mesos/conf/zk --quorum=file:///etc/mesos/conf/quorum
> Where zk=zk://zoo.service.consul:2181/mesos
>               quorum=2
>
>
>
> *Mesos-Slave Configuration:*
>
> mesos slave --work_dir=/var/lib/mesos-slave --ip=10.1.0.72
> --hostname=10.1.0.72 --strict=false  --master=
> file:///etc/mesos/conf/master FrameworkInfo.checkpoint=True
>
>
>
> *Marathon Configuration:*
>
> java -jar /opt/marathon.jar  --master zk://zoo.service.consul:2181/mesos
> --zk zk://zoo.service.consul:2181/marathon  --ha --hostname 10.1.0.72
> --checkpoint
>
>
>
>
> *Zookeeper configs with java version *"1.8.0_45"*:*
>
>
>
> dataDir=/var/lib/zookeeper
> clientPort=2181
> tickTime=2000
> initLimit=10
> syncLimit=20
>
>
> autopurge.purgeInterval=0
>
>
> zookeeper.connection.timeout.ms=6000
> server.1=10.1.0.70:2888:3888
> server.2=10.1.0.71:2888:3888
> server.3=10.1.0.72:2888:3888
>
> And different *myid* has been given.
>
>
> Can Anyone Help!!!
>
>
>


-- 

https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links