You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2014/02/27 05:50:20 UTC

[jira] [Commented] (TEZ-898) Inner/outer join with 0 maps fail with 'All inputs are expected to ask for memory'

    [ https://issues.apache.org/jira/browse/TEZ-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914062#comment-13914062 ] 

Siddharth Seth commented on TEZ-898:
------------------------------------

This can occur in cases where the number of tasks is 0 - some of the I/Os do not request for any memory in such cases.

> Inner/outer join with 0 maps fail with 'All inputs are expected to ask for memory'
> ----------------------------------------------------------------------------------
>
>                 Key: TEZ-898
>                 URL: https://issues.apache.org/jira/browse/TEZ-898
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Yesha Vora
>            Assignee: Siddharth Seth
>
> Inner and outer job operation  with 0 maps fail with 'All inputs are expected to ask for memory' 
> Example: 
> hadoop  jar /usr/lib/tez/tez-mapreduce-examples.jar join -r 11 -joinOp inner /tmp/Input /tmp/Output
> hadoop  jar /usr/lib/tez/tez-mapreduce-examples.jar join -r 11 -joinOp outer /tmp/Input /tmp/Output
> ===========================================
> hadoop  jar /usr/lib/tez/tez-mapreduce-examples.jar join -r 11 -joinOp inner /tmp/input /tmp/output
> 14/02/26 09:38:15 INFO client.RMProxy: Connecting to ResourceManager at host/10.1.1.1:8032
> 14/02/26 09:38:15 INFO client.RMProxy: Connecting to ResourceManager at host/10.1.1.1:8032
> Job started: Wed Feb 26 09:38:16 UTC 2014
>  INFO client.RMProxy: Connecting to ResourceManager at host/10.1.1.1:8032
> 14/02/26 09:38:16 INFO client.RMProxy: Connecting to ResourceManager at host/10.1.1.1:8032
> 14/02/26 09:38:16 INFO input.FileInputFormat: Total input paths to process : 0
> 14/02/26 09:38:17 INFO mapreduce.JobSubmitter: number of splits:0
> 14/02/26 09:38:17 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393401746744_0132
> 14/02/26 09:38:17 INFO client.YARNRunner: Number of stages: 2
> 14/02/26 09:38:17 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1393401746744_0132
> 14/02/26 09:38:17 INFO impl.YarnClientImpl: Submitted application application_1393401746744_0132
> 14/02/26 09:38:17 INFO client.RMProxy: Connecting to ResourceManager at host/10.1.1.1:8032
> 14/02/26 09:38:17 INFO client.RMProxy: Connecting to ResourceManager at host/10.1.1.1:8032
> 14/02/26 09:38:17 INFO mapreduce.Job: The url to track the job: http://host:8088/proxy/application_1393401746744_0132/
> 14/02/26 09:38:17 INFO mapreduce.Job: Running job: job_1393401746744_0132
> 14/02/26 09:38:22 INFO mapreduce.Job: Job job_1393401746744_0132 running in uber mode : false
> 14/02/26 09:38:22 INFO mapreduce.Job:  map 0% reduce 0%
> 14/02/26 09:38:33 INFO mapreduce.Job: Job job_1393401746744_0132 failed with state FAILED due to: Vertex failed, vertexName=finalreduce, vertexId=vertex_1393401746744_0132_1_01, diagnostics=[Task failed, taskId=task_1393401746744_0132_1_01_000000, diagnostics=[AttemptID:attempt_1393401746744_0132_1_01_000000_0 Info:Error: java.lang.IllegalStateException: All inputs are expected to ask for memory
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.tez.runtime.common.resources.MemoryDistributor.makeInitialAllocations(MemoryDistributor.java:124)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:224)
> at org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:522)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:515)
> Container released by application, AttemptID:attempt_1393401746744_0132_1_01_000000_1 Info:Error: java.lang.IllegalStateException: All inputs are expected to ask for memory
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.tez.runtime.common.resources.MemoryDistributor.makeInitialAllocations(MemoryDistributor.java:124)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:224)
> at org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:522)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:515)
> Container released by application, AttemptID:attempt_1393401746744_0132_1_01_000000_2 Info:Error: java.lang.IllegalStateException: All inputs are expected to ask for memory
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.tez.runtime.common.resources.MemoryDistributor.makeInitialAllocations(MemoryDistributor.java:124)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:224)
> at org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:522)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:515)
> Container released by application, AttemptID:attempt_1393401746744_0132_1_01_000000_3 Info:Error: java.lang.IllegalStateException: All inputs are expected to ask for memory
> at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
> at org.apache.tez.runtime.common.resources.MemoryDistributor.makeInitialAllocations(MemoryDistributor.java:124)
> at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:224)
> at org.apache.hadoop.mapred.YarnTezDagChild$4.run(YarnTezDagChild.java:522)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnTezDagChild.main(YarnTezDagChild.java:515)
> ], Vertex failed as one or more tasks failed. failedTasks:1]. DAG failed due to vertex failure. failedVertices:1 killedVertices:0
> 14/02/26 09:38:33 INFO mapreduce.Job: Counters: 0
> Job ended: Wed Feb 26 09:38:33 UTC 2014
> The job took 16 seconds.
> =========================================== 



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)