You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Szilard Nemeth (Jira)" <ji...@apache.org> on 2022/05/11 12:42:00 UTC
[jira] [Commented] (YARN-10850) TimelineService v2 lists containers for all attempts when filtering for one
[ https://issues.apache.org/jira/browse/YARN-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17534871#comment-17534871 ]
Szilard Nemeth commented on YARN-10850:
---------------------------------------
Hi [~bteke],
Just committed your PR to trunk.
Could you please check if this should be backported to older branches?
Thanks,
> TimelineService v2 lists containers for all attempts when filtering for one
> ---------------------------------------------------------------------------
>
> Key: YARN-10850
> URL: https://issues.apache.org/jira/browse/YARN-10850
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelinereader
> Reporter: Benjamin Teke
> Assignee: Benjamin Teke
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> When using the command
> {code:java}
> yarn container -list <app_attempt_id>
> {code}
> with an application attempt ID based on the help only the containers for that attempt should be listed.
> {code:java}
> -list <Application Name or Attempt ID> List containers for application
> attempt when application
> attempt ID is provided. When
> application name is provided,
> then it finds the instances of
> the application based on app's
> own implementation, and
> -appTypes option must be
> specified unless it is the
> default yarn-service type. With
> app name, it supports optional
> use of -version to filter
> instances based on app version,
> -components to filter instances
> based on component names,
> -states to filter instances
> based on instance state.
> {code}
> When TimelineService v2 is enabled all of the containers for the application are returned.
> {code:java}
> hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn applicationattempt -list application_1625124233002_0007
> 21/07/01 09:32:23 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
> 21/07/01 09:32:24 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
> 21/07/01 09:32:24 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
> Total number of application attempts :2
> ApplicationAttempt-Id State AM-Container-Id Tracking-URL
> appattempt_1625124233002_0007_000001 FAILED container_e43_1625124233002_0007_01_000001 http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/
> appattempt_1625124233002_0007_000002 KILLED container_e43_1625124233002_0007_02_000001 http://ctr-e172-1620330694487-146061-01-000003.hwx.site:8088/proxy/application_1625124233002_0007/
> {code}
> Querying the 2 app attempts produces the same output:
> {code:java}
> hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000001
> 21/07/01 09:32:35 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
> 21/07/01 09:32:35 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
> 21/07/01 09:32:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
> 21/07/01 09:32:36 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
> Total number of containers :12
> Container-Id Start Time Finish Time State Host Node Http Address LOG-URL
> container_e43_1625124233002_0007_02_000004 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
> container_e43_1625124233002_0007_02_000005 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
> container_e43_1625124233002_0007_02_000003 N/A N/A null ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
> container_e43_1625124233002_0007_02_000002 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
> container_e43_1625124233002_0007_02_000006 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
> container_e43_1625124233002_0007_02_000001 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
> container_e43_1625124233002_0007_01_000005 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
> container_e43_1625124233002_0007_01_000004 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
> container_e43_1625124233002_0007_01_000002 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
> container_e43_1625124233002_0007_01_000006 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
> container_e43_1625124233002_0007_01_000003 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
> container_e43_1625124233002_0007_01_000001 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa
> {code}
> {code:java}
> hrt_qa@ctr-e172-1620330694487-146061-01-000002:/hwqe/hadoopqe$ yarn container -list appattempt_1625124233002_0007_000002
> 21/07/01 09:32:50 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://ctr-e172-1620330694487-146061-01-000004.hwx.site:8198/ws/v2/timeline/, clusterId=yarn-cluster
> 21/07/01 09:32:51 INFO client.AHSProxy: Connecting to Application History server at ctr-e172-1620330694487-146061-01-000004.hwx.site/172.27.113.4:10200
> 21/07/01 09:32:51 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
> 21/07/01 09:32:51 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/7.1.7.0-504/0/resource-types.xml
> Total number of containers :12
> Container-Id Start Time Finish Time State Host Node Http Address LOG-URL
> container_e43_1625124233002_0007_02_000004 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_02_000004/container_e43_1625124233002_0007_02_000004/hrt_qa
> container_e43_1625124233002_0007_02_000005 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_02_000005/container_e43_1625124233002_0007_02_000005/hrt_qa
> container_e43_1625124233002_0007_02_000003 N/A N/A null ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000003/container_e43_1625124233002_0007_02_000003/hrt_qa
> container_e43_1625124233002_0007_02_000002 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_02_000002/container_e43_1625124233002_0007_02_000002/hrt_qa
> container_e43_1625124233002_0007_02_000006 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_02_000006/container_e43_1625124233002_0007_02_000006/hrt_qa
> container_e43_1625124233002_0007_02_000001 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_02_000001/container_e43_1625124233002_0007_02_000001/hrt_qa
> container_e43_1625124233002_0007_01_000005 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000002.hwx.site:25454 ctr-e172-1620330694487-146061-01-000002.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000002.hwx.site:25454/container_e43_1625124233002_0007_01_000005/container_e43_1625124233002_0007_01_000005/hrt_qa
> container_e43_1625124233002_0007_01_000004 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000004/container_e43_1625124233002_0007_01_000004/hrt_qa
> container_e43_1625124233002_0007_01_000002 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000005.hwx.site:25454 ctr-e172-1620330694487-146061-01-000005.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000005.hwx.site:25454/container_e43_1625124233002_0007_01_000002/container_e43_1625124233002_0007_01_000002/hrt_qa
> container_e43_1625124233002_0007_01_000006 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000006.hwx.site:25454 ctr-e172-1620330694487-146061-01-000006.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000006.hwx.site:25454/container_e43_1625124233002_0007_01_000006/container_e43_1625124233002_0007_01_000006/hrt_qa
> container_e43_1625124233002_0007_01_000003 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000007.hwx.site:25454 ctr-e172-1620330694487-146061-01-000007.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000007.hwx.site:25454/container_e43_1625124233002_0007_01_000003/container_e43_1625124233002_0007_01_000003/hrt_qa
> container_e43_1625124233002_0007_01_000001 N/A N/A COMPLETE ctr-e172-1620330694487-146061-01-000004.hwx.site:25454 ctr-e172-1620330694487-146061-01-000004.hwx.site:8042 http://ctr-e172-1620330694487-146061-01-000004.hwx.site:19888/jobhistory/logs/logs/ctr-e172-1620330694487-146061-01-000004.hwx.site:25454/container_e43_1625124233002_0007_01_000001/container_e43_1625124233002_0007_01_000001/hrt_qa
> {code}
> The reason for this is that the [container collector method|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AHSv2ClientImpl.java#L135] in TimelineServiceV2 gets the *application ID* from the *attempt ID* and collects the [containers for the application|https://github.com/apache/hadoop/blob/a3edfddcf7822ea13bdf4858672eb82cea5e0b5f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/TimelineReaderClient.java#L116].
> The other implementations however are using the application attempt ID: [ApplicationHistoryManagerOnTimelineStore|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerOnTimelineStore.java#L221] and [ApplicationHistoryManagerImpl|https://github.com/apache/hadoop/blob/1e3a6efcef2924a7966c44ca63476c853956691d/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/ApplicationHistoryManagerImpl.java#L235].
> TSv2 should behave similarly to v1.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org