You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Thomas Wang <w...@datability.io> on 2021/06/27 23:24:25 UTC
Yarn Application Crashed?
Hi,
I recently experienced a job crash due to the underlying Yarn application
failing for some reason. Here is the only error message I saw. It seems I
can no longer see any of the Flink job logs.
Application application_1623861596410_0010 failed 1 times (global limit =2;
local limit is =1) due to ApplicationMaster for attempt
appattempt_1623861596410_0010_000001 timed out. Failing the application.
I was running the Flink job using the Yarn session mode with the following
command.
export HADOOP_CLASSPATH=`hadoop classpath` &&
/usr/lib/flink/bin/yarn-session.sh -jm 7g -tm 7g -s 4 --detached
I didn't have HA setup, but I believe the underlying Yarn application
caused the crash because if, for some reason, the Flink job failed, the
Yarn application should still survive. Please correct me if this is not the
right assumption.
My question is how I should find the root cause in this case and what's the
recommended way to avoid this going forward?
Thanks.
Thomas
Re: Yarn Application Crashed?
Posted by Piotr Nowojski <pn...@apache.org>.
You are welcome :)
Piotrek
śr., 30 cze 2021 o 08:34 Thomas Wang <w...@datability.io> napisał(a):
> Thanks Piotr. This is helpful.
>
> Thomas
>
> On Mon, Jun 28, 2021 at 8:29 AM Piotr Nowojski <pn...@apache.org>
> wrote:
>
>> Hi,
>>
>> You should still be able to get the Flink logs via:
>>
>> > yarn logs -applicationId application_1623861596410_0010
>>
>> And it should give you more answers about what has happened.
>>
>> About the Flink and YARN behaviour, have you seen the documentation? [1]
>> Especially this part:
>>
>> > Failed containers (including the JobManager) are replaced by YARN. The
>> maximum number of JobManager container restarts is configured via
>> yarn.application-attempts (default 1). The YARN Application will fail once
>> all attempts are exhausted.
>>
>> ?
>>
>> Best,
>> Piotrek
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/yarn/#flink-on-yarn-reference
>>
>> pon., 28 cze 2021 o 02:26 Thomas Wang <w...@datability.io> napisał(a):
>>
>>> Just found some additional info. It looks like one of the EC2 instances
>>> got terminated at the time the crash happened and this job had 7 Task
>>> Managers running on that EC2 instance. Now I suspect it's possible
>>> that when Yarn tried to migrate the Task Managers, there were no idle
>>> containers as this job was using like 99% of the entire cluster. However in
>>> that case shouldn't Yarn wait for containers to become available? I'm not
>>> quite sure how Flink would behave in this case. Could someone provide some
>>> insights here? Thanks.
>>>
>>> Thomas
>>>
>>> On Sun, Jun 27, 2021 at 4:24 PM Thomas Wang <w...@datability.io> wrote:
>>>
>>>> Hi,
>>>>
>>>> I recently experienced a job crash due to the underlying Yarn
>>>> application failing for some reason. Here is the only error message I saw.
>>>> It seems I can no longer see any of the Flink job logs.
>>>>
>>>> Application application_1623861596410_0010 failed 1 times (global limit
>>>> =2; local limit is =1) due to ApplicationMaster for attempt
>>>> appattempt_1623861596410_0010_000001 timed out. Failing the application.
>>>>
>>>> I was running the Flink job using the Yarn session mode with the
>>>> following command.
>>>>
>>>> export HADOOP_CLASSPATH=`hadoop classpath` &&
>>>> /usr/lib/flink/bin/yarn-session.sh -jm 7g -tm 7g -s 4 --detached
>>>>
>>>> I didn't have HA setup, but I believe the underlying Yarn application
>>>> caused the crash because if, for some reason, the Flink job failed, the
>>>> Yarn application should still survive. Please correct me if this is not the
>>>> right assumption.
>>>>
>>>> My question is how I should find the root cause in this case and what's
>>>> the recommended way to avoid this going forward?
>>>>
>>>> Thanks.
>>>>
>>>> Thomas
>>>>
>>>
Re: Yarn Application Crashed?
Posted by Thomas Wang <w...@datability.io>.
Thanks Piotr. This is helpful.
Thomas
On Mon, Jun 28, 2021 at 8:29 AM Piotr Nowojski <pn...@apache.org> wrote:
> Hi,
>
> You should still be able to get the Flink logs via:
>
> > yarn logs -applicationId application_1623861596410_0010
>
> And it should give you more answers about what has happened.
>
> About the Flink and YARN behaviour, have you seen the documentation? [1]
> Especially this part:
>
> > Failed containers (including the JobManager) are replaced by YARN. The
> maximum number of JobManager container restarts is configured via
> yarn.application-attempts (default 1). The YARN Application will fail once
> all attempts are exhausted.
>
> ?
>
> Best,
> Piotrek
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/yarn/#flink-on-yarn-reference
>
> pon., 28 cze 2021 o 02:26 Thomas Wang <w...@datability.io> napisał(a):
>
>> Just found some additional info. It looks like one of the EC2 instances
>> got terminated at the time the crash happened and this job had 7 Task
>> Managers running on that EC2 instance. Now I suspect it's possible
>> that when Yarn tried to migrate the Task Managers, there were no idle
>> containers as this job was using like 99% of the entire cluster. However in
>> that case shouldn't Yarn wait for containers to become available? I'm not
>> quite sure how Flink would behave in this case. Could someone provide some
>> insights here? Thanks.
>>
>> Thomas
>>
>> On Sun, Jun 27, 2021 at 4:24 PM Thomas Wang <w...@datability.io> wrote:
>>
>>> Hi,
>>>
>>> I recently experienced a job crash due to the underlying Yarn
>>> application failing for some reason. Here is the only error message I saw.
>>> It seems I can no longer see any of the Flink job logs.
>>>
>>> Application application_1623861596410_0010 failed 1 times (global limit
>>> =2; local limit is =1) due to ApplicationMaster for attempt
>>> appattempt_1623861596410_0010_000001 timed out. Failing the application.
>>>
>>> I was running the Flink job using the Yarn session mode with the
>>> following command.
>>>
>>> export HADOOP_CLASSPATH=`hadoop classpath` &&
>>> /usr/lib/flink/bin/yarn-session.sh -jm 7g -tm 7g -s 4 --detached
>>>
>>> I didn't have HA setup, but I believe the underlying Yarn application
>>> caused the crash because if, for some reason, the Flink job failed, the
>>> Yarn application should still survive. Please correct me if this is not the
>>> right assumption.
>>>
>>> My question is how I should find the root cause in this case and what's
>>> the recommended way to avoid this going forward?
>>>
>>> Thanks.
>>>
>>> Thomas
>>>
>>
Re: Yarn Application Crashed?
Posted by Piotr Nowojski <pn...@apache.org>.
Hi,
You should still be able to get the Flink logs via:
> yarn logs -applicationId application_1623861596410_0010
And it should give you more answers about what has happened.
About the Flink and YARN behaviour, have you seen the documentation? [1]
Especially this part:
> Failed containers (including the JobManager) are replaced by YARN. The
maximum number of JobManager container restarts is configured via
yarn.application-attempts (default 1). The YARN Application will fail once
all attempts are exhausted.
?
Best,
Piotrek
[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/yarn/#flink-on-yarn-reference
pon., 28 cze 2021 o 02:26 Thomas Wang <w...@datability.io> napisał(a):
> Just found some additional info. It looks like one of the EC2 instances
> got terminated at the time the crash happened and this job had 7 Task
> Managers running on that EC2 instance. Now I suspect it's possible
> that when Yarn tried to migrate the Task Managers, there were no idle
> containers as this job was using like 99% of the entire cluster. However in
> that case shouldn't Yarn wait for containers to become available? I'm not
> quite sure how Flink would behave in this case. Could someone provide some
> insights here? Thanks.
>
> Thomas
>
> On Sun, Jun 27, 2021 at 4:24 PM Thomas Wang <w...@datability.io> wrote:
>
>> Hi,
>>
>> I recently experienced a job crash due to the underlying Yarn application
>> failing for some reason. Here is the only error message I saw. It seems I
>> can no longer see any of the Flink job logs.
>>
>> Application application_1623861596410_0010 failed 1 times (global limit
>> =2; local limit is =1) due to ApplicationMaster for attempt
>> appattempt_1623861596410_0010_000001 timed out. Failing the application.
>>
>> I was running the Flink job using the Yarn session mode with the
>> following command.
>>
>> export HADOOP_CLASSPATH=`hadoop classpath` &&
>> /usr/lib/flink/bin/yarn-session.sh -jm 7g -tm 7g -s 4 --detached
>>
>> I didn't have HA setup, but I believe the underlying Yarn application
>> caused the crash because if, for some reason, the Flink job failed, the
>> Yarn application should still survive. Please correct me if this is not the
>> right assumption.
>>
>> My question is how I should find the root cause in this case and what's
>> the recommended way to avoid this going forward?
>>
>> Thanks.
>>
>> Thomas
>>
>
Re: Yarn Application Crashed?
Posted by Thomas Wang <w...@datability.io>.
Just found some additional info. It looks like one of the EC2 instances got
terminated at the time the crash happened and this job had 7 Task Managers
running on that EC2 instance. Now I suspect it's possible that when Yarn
tried to migrate the Task Managers, there were no idle containers as this
job was using like 99% of the entire cluster. However in that case
shouldn't Yarn wait for containers to become available? I'm not quite sure
how Flink would behave in this case. Could someone provide some insights
here? Thanks.
Thomas
On Sun, Jun 27, 2021 at 4:24 PM Thomas Wang <w...@datability.io> wrote:
> Hi,
>
> I recently experienced a job crash due to the underlying Yarn application
> failing for some reason. Here is the only error message I saw. It seems I
> can no longer see any of the Flink job logs.
>
> Application application_1623861596410_0010 failed 1 times (global limit
> =2; local limit is =1) due to ApplicationMaster for attempt
> appattempt_1623861596410_0010_000001 timed out. Failing the application.
>
> I was running the Flink job using the Yarn session mode with the following
> command.
>
> export HADOOP_CLASSPATH=`hadoop classpath` &&
> /usr/lib/flink/bin/yarn-session.sh -jm 7g -tm 7g -s 4 --detached
>
> I didn't have HA setup, but I believe the underlying Yarn application
> caused the crash because if, for some reason, the Flink job failed, the
> Yarn application should still survive. Please correct me if this is not the
> right assumption.
>
> My question is how I should find the root cause in this case and what's
> the recommended way to avoid this going forward?
>
> Thanks.
>
> Thomas
>