You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "yinghua_zh (Jira)" <ji...@apache.org> on 2020/06/16 14:50:00 UTC
[jira] [Updated] (YARN-10317) RM returns a negative value when TEZ
AM requests resources
[ https://issues.apache.org/jira/browse/YARN-10317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
yinghua_zh updated YARN-10317:
------------------------------
Description:
RM returns a negative value when TEZ AM requests resources,The records are as follows：
2020-06-16 15:10:15,726 [INFO] [IPC Server listener on 23482] |ipc.Server|: IPC Server listener on 23482: starting
2020-06-16 15:10:15,726 [INFO] [ServiceThread:DAGClientRPCServer] |client.DAGClientServer|: Instantiated DAGClientRPCServer at sdp-10-88-0-19/10.88.0.19:23482
2020-06-16 15:10:15,726 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
2020-06-16 15:10:15,730 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2020-06-16 15:10:15,734 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: adding path spec: /*
2020-06-16 15:10:15,954 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Registered webapp guice modules
2020-06-16 15:10:15,955 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Jetty bound to port 28343
2020-06-16 15:10:15,956 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: jetty-6.1.26
2020-06-16 15:10:15,979 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Extract jar:[file:/data/data6/yarn/local/filecache/17/tez.tar.gz/lib/hadoop-yarn-common-2.7.2.jar!/webapps/|file://data/data6/yarn/local/filecache/17/tez.tar.gz/lib/hadoop-yarn-common-2.7.2-SDP.jar!/webapps/] to /data/data1/yarn/local/usercache/zyh/appcache/application_1592291210011_0010/container_e13_1592291210011_0010_01_000001/tmp/Jetty_0_0_0_0_28343_webapps____mdg1c9/webapp
2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:28343
2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Web app started at 28343
2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |web.WebUIService|: Instantiated WebUIService at [http://10-88-0-19:28343/ui/|http://sdp-10-88-0-19:28343/ui/]
2020-06-16 15:10:16,125 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: YarnTaskSchedulerService
2020-06-16 15:10:16,148 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2020-06-16 15:10:16,149 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=111101111
2020-06-16 15:10:16,159 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 250, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 10000, idleContainerMaxTimeout: 20000, sessionMinHeldContainers: 0
2020-06-16 15:10:16,235 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1592291210011_0010_000001, startTime=1592291416235
2020-06-16 15:10:16,235 [INFO] [main] |app.DAGAppMaster|: In Session mode. Waiting for DAG over RPC
2020-06-16 15:10:16,261 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: App total resource memory: -2048 cpu: 0 taskAllocations: 0
2020-06-16 15:10:16,262 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: {color:#ff0000}*A**llocated: <memory:0, vCores:0> Free: <memory:-2048, vCores:0>*{color} pendingRequests: 0 delayedContainers: 0 heartbeats: 1 lastPreemptionHeartbeat: 0
2020-06-16 15:10:16,264 [INFO] [Dispatcher thread \\{Central}] |node.PerSourceNodeTracker|: Num cluster nodes = 11
This leads to errors in tez segmentation
was:
RM returns a negative value when TEZ AM requests resources,The records are as follows：
2020-06-16 15:10:15,726 [INFO] [IPC Server listener on 23482] |ipc.Server|: IPC Server listener on 23482: starting
2020-06-16 15:10:15,726 [INFO] [ServiceThread:DAGClientRPCServer] |client.DAGClientServer|: Instantiated DAGClientRPCServer at sdp-10-88-0-19/10.88.0.19:23482
2020-06-16 15:10:15,726 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
2020-06-16 15:10:15,730 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2020-06-16 15:10:15,734 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: adding path spec: /*
2020-06-16 15:10:15,954 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Registered webapp guice modules
2020-06-16 15:10:15,955 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Jetty bound to port 28343
2020-06-16 15:10:15,956 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: jetty-6.1.26
2020-06-16 15:10:15,979 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Extract jar:file:/data/data6/yarn/local/filecache/17/tez.tar.gz/lib/hadoop-yarn-common-2.7.2-SDP.jar!/webapps/ to /data/data1/yarn/local/usercache/zyh/appcache/application_1592291210011_0010/container_e13_1592291210011_0010_01_000001/tmp/Jetty_0_0_0_0_28343_webapps____mdg1c9/webapp
2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:28343
2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Web app started at 28343
2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |web.WebUIService|: Instantiated WebUIService at http://sdp-10-88-0-19:28343/ui/
2020-06-16 15:10:16,125 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: YarnTaskSchedulerService
2020-06-16 15:10:16,148 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2020-06-16 15:10:16,149 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=111101111
2020-06-16 15:10:16,159 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 250, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 10000, idleContainerMaxTimeout: 20000, sessionMinHeldContainers: 0
2020-06-16 15:10:16,235 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1592291210011_0010_000001, startTime=1592291416235
2020-06-16 15:10:16,235 [INFO] [main] |app.DAGAppMaster|: In Session mode. Waiting for DAG over RPC
2020-06-16 15:10:16,261 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: App total resource memory: -2048 cpu: 0 taskAllocations: 0
2020-06-16 15:10:16,262 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: {color:#FF0000}*A**llocated: <memory:0, vCores:0> Free: <memory:-2048, vCores:0>*{color} pendingRequests: 0 delayedContainers: 0 heartbeats: 1 lastPreemptionHeartbeat: 0
2020-06-16 15:10:16,264 [INFO] [Dispatcher thread \{Central}] |node.PerSourceNodeTracker|: Num cluster nodes = 11
This leads to errors in tez segmentation
> RM returns a negative value when TEZ AM requests resources
> ----------------------------------------------------------
>
> Key: YARN-10317
> URL: https://issues.apache.org/jira/browse/YARN-10317
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
> Affects Versions: 2.7.2
> Reporter: yinghua_zh
> Priority: Major
>
> RM returns a negative value when TEZ AM requests resources,The records are as follows：
> 2020-06-16 15:10:15,726 [INFO] [IPC Server listener on 23482] |ipc.Server|: IPC Server listener on 23482: starting
> 2020-06-16 15:10:15,726 [INFO] [ServiceThread:DAGClientRPCServer] |client.DAGClientServer|: Instantiated DAGClientRPCServer at sdp-10-88-0-19/10.88.0.19:23482
> 2020-06-16 15:10:15,726 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
> 2020-06-16 15:10:15,730 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
> 2020-06-16 15:10:15,734 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: adding path spec: /*
> 2020-06-16 15:10:15,954 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Registered webapp guice modules
> 2020-06-16 15:10:15,955 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |http.HttpServer2|: Jetty bound to port 28343
> 2020-06-16 15:10:15,956 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: jetty-6.1.26
> 2020-06-16 15:10:15,979 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Extract jar:[file:/data/data6/yarn/local/filecache/17/tez.tar.gz/lib/hadoop-yarn-common-2.7.2.jar!/webapps/|file://data/data6/yarn/local/filecache/17/tez.tar.gz/lib/hadoop-yarn-common-2.7.2-SDP.jar!/webapps/] to /data/data1/yarn/local/usercache/zyh/appcache/application_1592291210011_0010/container_e13_1592291210011_0010_01_000001/tmp/Jetty_0_0_0_0_28343_webapps____mdg1c9/webapp
> 2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:28343
> 2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Web app started at 28343
> 2020-06-16 15:10:16,123 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |web.WebUIService|: Instantiated WebUIService at [http://10-88-0-19:28343/ui/|http://sdp-10-88-0-19:28343/ui/]
> 2020-06-16 15:10:16,125 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: YarnTaskSchedulerService
> 2020-06-16 15:10:16,148 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
> 2020-06-16 15:10:16,149 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating TaskScheduler: Local TaskScheduler with clusterIdentifier=111101111
> 2020-06-16 15:10:16,159 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 250, containerReuseEnabled: true, reuseRackLocal: true, reuseNonLocal: false, localitySchedulingDelay: 250, preemptionPercentage: 10, preemptionMaxWaitTime: 60000, numHeartbeatsBetweenPreemptions: 3, idleContainerMinTimeout: 10000, idleContainerMaxTimeout: 20000, sessionMinHeldContainers: 0
> 2020-06-16 15:10:16,235 [INFO] [main] |history.HistoryEventHandler|: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1592291210011_0010_000001, startTime=1592291416235
> 2020-06-16 15:10:16,235 [INFO] [main] |app.DAGAppMaster|: In Session mode. Waiting for DAG over RPC
> 2020-06-16 15:10:16,261 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: App total resource memory: -2048 cpu: 0 taskAllocations: 0
> 2020-06-16 15:10:16,262 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: {color:#ff0000}*A**llocated: <memory:0, vCores:0> Free: <memory:-2048, vCores:0>*{color} pendingRequests: 0 delayedContainers: 0 heartbeats: 1 lastPreemptionHeartbeat: 0
> 2020-06-16 15:10:16,264 [INFO] [Dispatcher thread \\{Central}] |node.PerSourceNodeTracker|: Num cluster nodes = 11
> This leads to errors in tez segmentation
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org