You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun Prakash (Created) (JIRA)" <ji...@apache.org> on 2012/01/09 17:26:39 UTC
[jira] [Created] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Reducer Job always goes to pending state.
-----------------------------------------
Key: MAPREDUCE-3643
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun Prakash
Priority: Minor
I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
I have ran this job several times there was no issues but i am facing the problem right now.
Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182616#comment-13182616 ]
Robert Joseph Evans commented on MAPREDUCE-3643:
------------------------------------------------
Is there any more information you can give on this? How reproducible is it? It sounds like it is kind of sporadic. So the reducer is the identity reducer, what about the mapper. Do you any logs that you can upload. The JT log for the time of the error would be very helpful.
> Reducer Job always goes to pending state.
> -----------------------------------------
>
> Key: MAPREDUCE-3643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Prakash
> Priority: Minor
>
> I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
> In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
> I have ran this job several times there was no issues but i am facing the problem right now.
> Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Posted by "Arun Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182631#comment-13182631 ]
Arun Prakash commented on MAPREDUCE-3643:
-----------------------------------------
I have attached the snapshot for the job tracker and below i mentioned the console output.
12/01/09 11:36:29 INFO streaming.StreamJob: map 0% reduce 0%
12/01/09 11:36:34 INFO streaming.StreamJob: map 100% reduce 0%
Streaming command for step4
hadoop fs -rmr $OUTPUT
hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-*.jar \
-D mapred.job.name="$NAME" \
-D mapred.reduce.tasks=1 \
-D mapred.task.timeout=$TIMEOUT \
-input $INPUT1 \
-input $INPUT2 \
-mapper "/bin/awk 'gsub(/\t/, \",\")'" \
-reducer "/bin/cat" \
-output $OUTPUT
i have used the /bin/cat instead of identity reducer and mapper
> Reducer Job always goes to pending state.
> -----------------------------------------
>
> Key: MAPREDUCE-3643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Prakash
> Priority: Minor
> Attachments: snap.JPG
>
>
> I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
> In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
> I have ran this job several times there was no issues but i am facing the problem right now.
> Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Posted by "Arun Prakash (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun Prakash updated MAPREDUCE-3643:
------------------------------------
Attachment: snap.JPG
> Reducer Job always goes to pending state.
> -----------------------------------------
>
> Key: MAPREDUCE-3643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Prakash
> Priority: Minor
> Attachments: snap.JPG
>
>
> I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
> In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
> I have ran this job several times there was no issues but i am facing the problem right now.
> Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182633#comment-13182633 ]
Robert Joseph Evans commented on MAPREDUCE-3643:
------------------------------------------------
This is unrelated, but for your information having {code}-reducer NONE{code} or not including a -reducer command is going to be faster then using {code}-reducer /bin/cat{code} as the streaming code will not bother to stream the data through cat which will reduce the IO and processing needed.
> Reducer Job always goes to pending state.
> -----------------------------------------
>
> Key: MAPREDUCE-3643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Prakash
> Priority: Minor
> Attachments: snap.JPG
>
>
> I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
> In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
> I have ran this job several times there was no issues but i am facing the problem right now.
> Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182630#comment-13182630 ]
Robert Joseph Evans commented on MAPREDUCE-3643:
------------------------------------------------
Is it stuck in pending for a very long period of time? Have you checked the queue that you submitted to, to see if it is full or not?
> Reducer Job always goes to pending state.
> -----------------------------------------
>
> Key: MAPREDUCE-3643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Prakash
> Priority: Minor
> Attachments: snap.JPG
>
>
> I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
> In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
> I have ran this job several times there was no issues but i am facing the problem right now.
> Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Posted by "Arun Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182613#comment-13182613 ]
Arun Prakash commented on MAPREDUCE-3643:
-----------------------------------------
I am using the hadoop version 0.20,i couldn't find the exact version in the field.
> Reducer Job always goes to pending state.
> -----------------------------------------
>
> Key: MAPREDUCE-3643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Prakash
> Priority: Minor
>
> I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
> In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
> I have ran this job several times there was no issues but i am facing the problem right now.
> Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3643) Reducer Job always goes to
pending state.
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182608#comment-13182608 ]
Robert Joseph Evans commented on MAPREDUCE-3643:
------------------------------------------------
What version is this filed against? Can you update the affects version field?
> Reducer Job always goes to pending state.
> -----------------------------------------
>
> Key: MAPREDUCE-3643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Arun Prakash
> Priority: Minor
>
> I am using hadoop streaming command for map-reduce.In my job there are 4 streaming commands,i have no issues with first 3 steps but step4 job once mapper got 100% completed directly going to pending state,even there was no error in the job tracker.
> In the step4,i am merging the output(i.e step2 and step 3 output) as single output using identity reducer.
> I have ran this job several times there was no issues but i am facing the problem right now.
> Any suggestions!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira