You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Shaulboo Zhiao (JIRA)" <ji...@apache.org> on 2016/12/02 08:39:59 UTC
[jira] [Created] (YARN-5960) yarn capacity
DominantResourceCalculator can work on mapreduce , but got error on tez
Shaulboo Zhiao created YARN-5960:
------------------------------------
Summary: yarn capacity DominantResourceCalculator can work on mapreduce , but got error on tez
Key: YARN-5960
URL: https://issues.apache.org/jira/browse/YARN-5960
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler, yarn
Affects Versions: 2.7.1
Environment: HDP 2.4.2.0-258 , hadoop 2.7.1 , hive 1.2 , tez 0.7.0 , ranger 0.5.0
Reporter: Shaulboo Zhiao
Priority: Minor
Yestoday ,I configured a queue with 30% resources , 24core CPU & 1000G memory . Until then everything is OK，while hql runs over tez engine, ranger-hive is on and yarn uses capacity scheduler . After I configured 'yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator' , hql executed failed .The message below :
INFO : Session is already open
INFO : Dag name: select local_code , count(1) fr...local_code(Stage-1)
INFO : Tez session was closed. Reopening...
INFO : Session re-established.
INFO :
INFO : Status: Running (Executing on YARN cluster with App id application_1480664840037_0003)
INFO : Map 1: -/- Reducer 2: 0/14
INFO : Map 1: 0/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-2)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-4)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-5)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-16)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-35)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-38)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-44)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-50)/36 Reducer 2: 0/14
INFO : Map 1: 0(+1,-56)/36 Reducer 2: 0/14
INFO : Map 1: 0(+1,-59)/36 Reducer 2: 0/14
INFO : Map 1: 0(+2,-63)/36 Reducer 2: 0/14
INFO : Map 1: 0(+2,-70)/36 Reducer 2: 0/14
INFO : Map 1: 0(+5,-77)/36 Reducer 2: 0/14
INFO : Map 1: 0(+0,-84)/36 Reducer 2: 0/14
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1480664840037_0003_1_00, diagnostics=[Task failed, taskId=task_1480664840037_0003_1_00_000029, diagnostics=[TaskAttempt 0 failed, info=[Container container_e25_1480664840037_0003_01_000034 finished with diagnostics set to [Container failed, exitCode=-1000. Application application_1480664840037_0003 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is nobody
main : requested yarn user is hive
Requested user nobody is not whitelisted and has id 99,which is below the minimum allowed 500
It seems that yarn run as user hive before , because the ranger is on and ask user hive to execut all hql . But I set capacity resouece calculator to 'org.apache.hadoop.yarn.util.resource.DominantResourceCalculator' , Yarn run task with user 'NOBODY' , that make the hive run failed everytime . I change hive execution engine to mr , everything is OK . As you know, I need tez to improve performance over mapreduce , so is it the yarn capacity problem.
Feedback would be appreciated!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org