You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Prashant Nayak <pr...@intellifylearning.com> on 2017/07/12 18:09:53 UTC
Flink Mesos Outstanding Offers - trouble launching task managers
Hi
We’re running Flink 1.3.1 on Mesos.
From time-to-time, the Flink app master seems to have trouble with Mesos
offers… At such time, it obviously ends up not launching the requested task
managers (mesos.initial-tasks) and we’ve noticed situations where it
launches zero tasks. During such
times we see a long list of “Outstanding Offers” in the Mesos UI. At the
same time, the app master logs have the following
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.mesos.scheduler.LaunchCoordinator - Processing 12
task(s) against 0 new offer(s) plus outstanding offers.
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.mesos.scheduler.LaunchCoordinator - Resources
considered: (note: expired offers not deducted from below)
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.mesos.scheduler.LaunchCoordinator - 10.80.xx.6 has 0.0
MB, 0.0 cpus
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.mesos.scheduler.LaunchCoordinator - 10.80.xx.233 has
0.0 MB, 0.0 cpus
2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
org.apache.flink.mesos.scheduler.LaunchCoordinator - Waiting for more
offers; 12 task(s) are not yet launched.
The two Mesos agents above (10.80.xx.6, 10.80.xx.233) are listed as having
offers outstanding to the Flink framework in the Mesos UI
Appreciate any input on how to go about resolving such an issue.
Thanks
Prashant
Re: Flink Mesos Outstanding Offers - trouble launching task managers
Posted by Prashant Nayak <pr...@intellifylearning.com>.
Thanks… will do this early next week. Appreciate the follow-up.
Prashant
On Thu, Aug 31, 2017 at 5:45 PM, Eron Wright <er...@gmail.com> wrote:
> Please mail me more information, in particular the JM log and the
> information on the 'offers' tab on the Mesos UI. Also, are you using any
> Mesos roles?
>
> Thanks
>
> On Thu, Aug 31, 2017 at 9:02 AM, prashantnayak <
> prashant@intellifylearning.com> wrote:
>
>> Hi Eron
>>
>> No, unfortunately we did not directly resolve it... we work around it for
>> now by ensuring that our Mesos slaves are set up to correctly support the
>> JobManager with offers.
>>
>> Prashant
>>
>>
>>
>> --
>> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/
>>
>
>
--
Prashant Nayak
CTO / Founder - Intellify Learning
m: 508.410.5380 | o: 617.701.6330 x701| skype: prashant.nayak.skype
Re: Flink Mesos Outstanding Offers - trouble launching task managers
Posted by Eron Wright <er...@gmail.com>.
Please mail me more information, in particular the JM log and the
information on the 'offers' tab on the Mesos UI. Also, are you using any
Mesos roles?
Thanks
On Thu, Aug 31, 2017 at 9:02 AM, prashantnayak <
prashant@intellifylearning.com> wrote:
> Hi Eron
>
> No, unfortunately we did not directly resolve it... we work around it for
> now by ensuring that our Mesos slaves are set up to correctly support the
> JobManager with offers.
>
> Prashant
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/
>
Re: Flink Mesos Outstanding Offers - trouble launching task
managers
Posted by prashantnayak <pr...@intellifylearning.com>.
Hi Eron
No, unfortunately we did not directly resolve it... we work around it for
now by ensuring that our Mesos slaves are set up to correctly support the
JobManager with offers.
Prashant
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink Mesos Outstanding Offers - trouble launching task managers
Posted by Eron Wright <er...@gmail.com>.
Hello, did you resolve this issue?
Thanks,
Eron Wright
On Wed, Jul 12, 2017 at 11:09 AM, Prashant Nayak <
prashant@intellifylearning.com> wrote:
>
> Hi
>
> We’re running Flink 1.3.1 on Mesos.
>
> From time-to-time, the Flink app master seems to have trouble with Mesos
> offers… At such time, it obviously ends up not launching the requested task
> managers (mesos.initial-tasks) and we’ve noticed situations where it
> launches zero tasks. During such
> times we see a long list of “Outstanding Offers” in the Mesos UI. At the
> same time, the app master logs have the following
>
>
> 2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
> org.apache.flink.mesos.scheduler.LaunchCoordinator - Processing 12
> task(s) against 0 new offer(s) plus outstanding offers.
> 2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
> org.apache.flink.mesos.scheduler.LaunchCoordinator - Resources
> considered: (note: expired offers not deducted from below)
> 2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
> org.apache.flink.mesos.scheduler.LaunchCoordinator - 10.80.xx.6 has
> 0.0 MB, 0.0 cpus
> 2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
> org.apache.flink.mesos.scheduler.LaunchCoordinator - 10.80.xx.233 has
> 0.0 MB, 0.0 cpus
> 2017-07-12 18:06:23.939 [flink-akka.actor.default-dispatcher-20] INFO
> org.apache.flink.mesos.scheduler.LaunchCoordinator - Waiting for more
> offers; 12 task(s) are not yet launched.
>
> The two Mesos agents above (10.80.xx.6, 10.80.xx.233) are listed as having
> offers outstanding to the Flink framework in the Mesos UI
>
> Appreciate any input on how to go about resolving such an issue.
>
> Thanks
> Prashant
>