You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Oucema Bellagha <Ou...@hotmail.com> on 2017/09/04 11:54:09 UTC
Marathon application deployment get stuck on Waiting status
Hi folks,
I have a 3 nodes setup running Marathon, mesos-master,mesos-slave and Zookeeper with HA config enabled, then tested a deployment of simple hello app using mesos-execute and it's working as expected.
Now everything looks fine, so I connect to Marathon and deploy a simple app to test marathon: (echo "hello" >> /tmp/output.txt) but the application get sucked in "waiting" status.
what could be the problem preventing Marathon to use mesos resources for deployment ?
Logs from mesos-master:
I0904 11:23:27.064332 19769 master.cpp:2813] Received SUBSCRIBE call for framework 'marathon' at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324<http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
I0904 11:23:27.064623 19769 master.cpp:2890] Subscribing framework marathon with checkpointing enabled and capabilities [ PARTITION_AWARE ]
I0904 11:23:27.064669 19769 master.cpp:6272] Updating info for framework cb16118a-2257-4020-a907-63aa6294e11b-0000
I0904 11:23:27.064697 19769 master.cpp:2994] Framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324<http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/> failed over
I0904 11:23:27.065032 19770 hierarchical.cpp:342] Activated framework cb16118a-2257-4020-a907-63aa6294e11b-0000
I0904 11:23:27.065465 19770 master.cpp:7305] Sending 3 offers to framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324<http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
I0904 11:23:27.907865 19769 http.cpp:1115] HTTP GET for /files/read?_=1504517007920&jsonp=jQuery17109098185077823333_1504516979864&length=50000&offset=352538&path=%2Fmaster%2Flog from 192.168.40.1:53525<http://192.168.40.1:53525/> with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
I0904 11:23:28.916651 19768 http.cpp:1115] HTTP GET for /files/read?_=1504517008930&jsonp=jQuery17109098185077823333_1504516979865&length=50000&offset=353797&path=%2Fmaster%2Flog from 192.168.40.1:53525<http://192.168.40.1:53525/> with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
E0904 11:23:30.071293 19775 process.cpp:2450] Failed to shutdown socket with fd 39, address 192.168.40.159:58072<http://192.168.40.159:58072/>: Transport endpoint is not connected
I0904 11:23:30.073277 19768 master.cpp:1430] Framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324<http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/> disconnected
I0904 11:23:30.073307 19768 master.cpp:3160] Deactivating framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324<http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
I0904 11:23:30.073485 19768 master.cpp:3137] Disconnecting framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324<http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
I0904 11:23:30.073496 19768 master.cpp:1445] Giving framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324<http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/> 1weeks to failover
I0904 11:23:30.073519 19768 hierarchical.cpp:374] Deactivated framework cb16118a-2257-4020-a907-63aa6294e11b-0000
Passpes,
Regards
Re: Marathon application deployment get stuck on Waiting status
Posted by Kiril Menshikov <km...@gmail.com>.
Hi,
Marathon execute long-running tasks. If look closer, you command work perfect and finalized fast. Marathon try to restart process.
Just open your mesos UI, you can find a lot of complete tasks. Or you have some configuration issues. But you also need to look at agent logs as well.
-Kirils
> On Sep 4, 2017, at 14:54, Oucema Bellagha <Ou...@hotmail.com> wrote:
>
> Hi folks,
>
> I have a 3 nodes setup running Marathon, mesos-master,mesos-slave and Zookeeper with HA config enabled, then tested a deployment of simple hello app using mesos-execute and it's working as expected.
>
> Now everything looks fine, so I connect to Marathon and deploy a simple app to test marathon: (echo "hello" >> /tmp/output.txt) but the application get sucked in "waiting" status.
>
>
>
> what could be the problem preventing Marathon to use mesos resources for deployment ?
>
>
>
> Logs from mesos-master:
>
>
>
> I0904 11:23:27.064332 19769 master.cpp:2813] Received SUBSCRIBE call for framework 'marathon' at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324 <http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
> I0904 11:23:27.064623 19769 master.cpp:2890] Subscribing framework marathon with checkpointing enabled and capabilities [ PARTITION_AWARE ]
> I0904 11:23:27.064669 19769 master.cpp:6272] Updating info for framework cb16118a-2257-4020-a907-63aa6294e11b-0000
> I0904 11:23:27.064697 19769 master.cpp:2994] Framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324 <http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/> failed over
> I0904 11:23:27.065032 19770 hierarchical.cpp:342] Activated framework cb16118a-2257-4020-a907-63aa6294e11b-0000
> I0904 11:23:27.065465 19770 master.cpp:7305] Sending 3 offers to framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324 <http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
> I0904 11:23:27.907865 19769 http.cpp:1115] HTTP GET for /files/read?_=1504517007920&jsonp=jQuery17109098185077823333_1504516979864&length=50000&offset=352538&path=%2Fmaster%2Flog from 192.168.40.1:53525 <http://192.168.40.1:53525/> with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
> I0904 11:23:28.916651 19768 http.cpp:1115] HTTP GET for /files/read?_=1504517008930&jsonp=jQuery17109098185077823333_1504516979865&length=50000&offset=353797&path=%2Fmaster%2Flog from 192.168.40.1:53525 <http://192.168.40.1:53525/> with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'
> E0904 11:23:30.071293 19775 process.cpp:2450] Failed to shutdown socket with fd 39, address 192.168.40.159:58072 <http://192.168.40.159:58072/>: Transport endpoint is not connected
> I0904 11:23:30.073277 19768 master.cpp:1430] Framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324 <http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/> disconnected
> I0904 11:23:30.073307 19768 master.cpp:3160] Deactivating framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324 <http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
> I0904 11:23:30.073485 19768 master.cpp:3137] Disconnecting framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324 <http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/>
> I0904 11:23:30.073496 19768 master.cpp:1445] Giving framework cb16118a-2257-4020-a907-63aa6294e11b-0000 (marathon) at scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324 <http://scheduler-0340362b-0bb6-4fb8-8501-118d976e2cbd@192.168.40.156:36324/> 1weeks to failover
> I0904 11:23:30.073519 19768 hierarchical.cpp:374] Deactivated framework cb16118a-2257-4020-a907-63aa6294e11b-0000
>
> Passpes,
> Regards