You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Kang Lee (Jira)" <ji...@apache.org> on 2022/12/14 07:31:00 UTC
[jira] [Updated] (YARN-11396) Used resource of user may be incorrect when flink's job manager retry
[ https://issues.apache.org/jira/browse/YARN-11396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kang Lee updated YARN-11396:
----------------------------
Description:
Run flink job on YARN 2.10.1 using the capacity scheduler，used resource of user is incorrect when job manager failed and attempt.
Reproduce this issue:
1. Create a capacity_test queue. The queue resource is following:
{code:java}
Queue State: RUNNING
Used Capacity: <memory:4096, vCores:4> (84.7%)
Configured Capacity: <memory:0, vCores:0>
Configured Max Capacity: unlimited
Effective Capacity: <memory:20479, vCores:4> (4.0%)
Effective Max Capacity: <memory:512000, vCores:118> (100.0%)
Absolute Used Capacity: 3.4%
Absolute Configured Capacity: 4.0%
Absolute Configured Max Capacity: 100.0%
Used Resources: <memory:4096, vCores:4>
{code}
2. Sumbit a flink job to yarn with parallelism is 10 and contaianer resource is 1c 1024m.
{code:java}
flink run -m yarn-cluster -yjm 1024 -ytm 1024 -parallelism 10 -yqu capacity_test /cloud/service/flink/examples/streaming/WindowJoin.ja {code}
Becuase user's max resource of this queue is 4c, 10g, so this job only can runnning 5 containers, at this moment, used resource of this user is following
||User Name||Max Resource||Weight||Used Resource||Max AM Resource||Used AM Resource||Schedulable Apps||Non-Schedulable Apps||
|hadoop|*<memory:20480, vCores:4>*|1.0|<memory:5120, vCores:5>|<memory:10240, vCores:2>|<memory:2048, vCores:2>|2|
3. kill -9 the process of job manager, so this application of attempt will be removed by yarn, and the user will be remove form UserManager as well.
In method of LeafQueue#removeApplicationAttempt, when user's total applications is 0, the user will be remove from usersManager.
{code:java}
private void removeApplicationAttempt(
FiCaSchedulerApp application, String userName) {
try {
writeLock.lock();
//...
user.finishApplication(wasActive);
if (user.getTotalApplications() == 0) {
usersManager.removeUser(application.getUser());
}
//...
}{code}
4. A new job manager will be attempted , so the User object of hadoop will be recreate, and used resource of user is initialize to 0. As the same time, in flink job, the value of ApplicationSubmissionContextProto#keep_containers_across_application_attempts is true, old containers can still running, and this part of resources is not compute in recreated user. So used resource of user is incorrect and real used resource more than max resource can bu used，like this
!image-2022-12-14-14-37-09-463.png|width=1192,height=532!
was:
Run flink job on YARN 2.10.1 using the capacity scheduler，used resource of user is incorrect when job manager failed and attempt.
Reproduce this issue:
1. Create a capacity_test queue. The queue resource is following:
{code:java}
Queue State: RUNNING
Used Capacity: <memory:4096, vCores:4> (84.7%)
Configured Capacity: <memory:0, vCores:0>
Configured Max Capacity: unlimited
Effective Capacity: <memory:20479, vCores:4> (4.0%)
Effective Max Capacity: <memory:512000, vCores:118> (100.0%)
Absolute Used Capacity: 3.4%
Absolute Configured Capacity: 4.0%
Absolute Configured Max Capacity: 100.0%
Used Resources: <memory:4096, vCores:4>
{code}
2. Sumbit a flink job to yarn with parallelism is 10 and contaianer resource is 1c 1024m.
{code:java}
flink run -m yarn-cluster -yjm 1024 -ytm 1024 -parallelism 10 -yqu capacity_test /cloud/service/flink/examples/streaming/WindowJoin.ja {code}
Becuase user's max resource of this queue is 4c, 10g, so this job only can runnning 5 containers, at this moment, used resource of this user is following
||User Name||Max Resource||Weight||Used Resource||Max AM Resource||Used AM Resource||Schedulable Apps||Non-Schedulable Apps||
|hadoop|*<memory:20480, vCores:4>*|1.0|<memory:5120, vCores:5>|<memory:10240, vCores:2>|<memory:2048, vCores:2>|2|
3. kill -9 the process of job manager, so this application of attempt will be removed by yarn, and the user will be remove form UserManager as well.
In method of LeafQueue#removeApplicationAttempt, when user's total applications is 0, the user will be remove from usersManager.
{code:java}
private void removeApplicationAttempt(
FiCaSchedulerApp application, String userName) {
try {
writeLock.lock();
//...
user.finishApplication(wasActive);
if (user.getTotalApplications() == 0) {
usersManager.removeUser(application.getUser());
}
//...
}{code}
4. A new job manager will be attempted , so the User object of hadoop will be recreate, and used resource of user is initialize to 0. As the same time, in flink job, the value of ApplicationSubmissionContextProto#keep_containers_across_application_attempts is true, old containers can still running, and this part of resources is not compute in recreated user. So used resource of user is incorrect and real used resource more than max resource can bu used，like this
!image-2022-12-14-14-37-09-463.png|width=1192,height=532!
> Used resource of user may be incorrect when flink's job manager retry
> ----------------------------------------------------------------------
>
> Key: YARN-11396
> URL: https://issues.apache.org/jira/browse/YARN-11396
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.10.1
> Reporter: Kang Lee
> Priority: Minor
> Attachments: image-2022-12-14-14-37-09-463.png
>
>
> Run flink job on YARN 2.10.1 using the capacity scheduler，used resource of user is incorrect when job manager failed and attempt.
> Reproduce this issue:
> 1. Create a capacity_test queue. The queue resource is following:
> {code:java}
> Queue State: RUNNING
> Used Capacity: <memory:4096, vCores:4> (84.7%)
> Configured Capacity: <memory:0, vCores:0>
> Configured Max Capacity: unlimited
> Effective Capacity: <memory:20479, vCores:4> (4.0%)
> Effective Max Capacity: <memory:512000, vCores:118> (100.0%)
> Absolute Used Capacity: 3.4%
> Absolute Configured Capacity: 4.0%
> Absolute Configured Max Capacity: 100.0%
> Used Resources: <memory:4096, vCores:4>
> {code}
> 2. Sumbit a flink job to yarn with parallelism is 10 and contaianer resource is 1c 1024m.
> {code:java}
> flink run -m yarn-cluster -yjm 1024 -ytm 1024 -parallelism 10 -yqu capacity_test /cloud/service/flink/examples/streaming/WindowJoin.ja {code}
> Becuase user's max resource of this queue is 4c, 10g, so this job only can runnning 5 containers, at this moment, used resource of this user is following
> ||User Name||Max Resource||Weight||Used Resource||Max AM Resource||Used AM Resource||Schedulable Apps||Non-Schedulable Apps||
> |hadoop|*<memory:20480, vCores:4>*|1.0|<memory:5120, vCores:5>|<memory:10240, vCores:2>|<memory:2048, vCores:2>|2|
> 3. kill -9 the process of job manager, so this application of attempt will be removed by yarn, and the user will be remove form UserManager as well.
> In method of LeafQueue#removeApplicationAttempt, when user's total applications is 0, the user will be remove from usersManager.
> {code:java}
> private void removeApplicationAttempt(
> FiCaSchedulerApp application, String userName) {
> try {
> writeLock.lock();
> //...
> user.finishApplication(wasActive);
> if (user.getTotalApplications() == 0) {
> usersManager.removeUser(application.getUser());
> }
> //...
> }{code}
> 4. A new job manager will be attempted , so the User object of hadoop will be recreate, and used resource of user is initialize to 0. As the same time, in flink job, the value of ApplicationSubmissionContextProto#keep_containers_across_application_attempts is true, old containers can still running, and this part of resources is not compute in recreated user. So used resource of user is incorrect and real used resource more than max resource can bu used，like this
> !image-2022-12-14-14-37-09-463.png|width=1192,height=532!
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org