You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Yoram Arnon (JIRA)" <ji...@apache.org> on 2006/05/10 01:13:04 UTC
[jira] Created: (HADOOP-206) local map-reduce job limited to single
reducer
local map-reduce job limited to single reducer
----------------------------------------------
Key: HADOOP-206
URL: http://issues.apache.org/jira/browse/HADOOP-206
Project: Hadoop
Type: Bug
Components: mapred
Versions: 0.1.0, 0.2, 0.1.1
Environment: local job tracker
Reporter: Yoram Arnon
Priority: Minor
when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and the number of reduce tasks is set at 1.
This prevents me from locally debugging my partition function, which tries to partition based on the number of reduce tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-206) local map-reduce job limited to
single reducer
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12489938 ]
Runping Qi commented on HADOOP-206:
-----------------------------------
If LocalJobRunner does not force to set numReduceTasks to 1,
unit tests TestMapRed and TestRecordMR failed due to out of memory
in reduce:
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.j
ava:53)
at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:91)
at org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java
:1621)
at org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey
(SequenceFile.java:2573)
at org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.next(SequenceFile
.java:2358)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.getNext(ReduceTask
.java:207)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.next(ReduceTask.ja
va:170)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceT
ask.java:245)
at org.apache.hadoop.mapred.TestMapRed$RandomCheckReducer.reduce(TestMap
Red.java:155)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:1
54)
Plus, LocalJobRunning calls only one reducer with partition equal 0.
That part has to be fixed too.
> local map-reduce job limited to single reducer
> ----------------------------------------------
>
> Key: HADOOP-206
> URL: https://issues.apache.org/jira/browse/HADOOP-206
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.1.0, 0.1.1, 0.2.0
> Environment: local job tracker
> Reporter: Yoram Arnon
> Assigned To: Runping Qi
> Priority: Minor
> Fix For: 0.13.0
>
>
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to partition based on the number of reduce tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-206) local map-reduce job limited to single
reducer
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-206:
---------------------------------
Fix Version/s: 0.13.0
Assignee: Runping Qi (was: Owen O'Malley)
> local map-reduce job limited to single reducer
> ----------------------------------------------
>
> Key: HADOOP-206
> URL: https://issues.apache.org/jira/browse/HADOOP-206
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.1.0, 0.1.1, 0.2.0
> Environment: local job tracker
> Reporter: Yoram Arnon
> Assigned To: Runping Qi
> Priority: Minor
> Fix For: 0.13.0
>
>
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to partition based on the number of reduce tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-206) local map-reduce job limited to single
reducer
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-206:
---------------------------------
Fix Version/s: (was: 0.13.0)
0.14.0
> local map-reduce job limited to single reducer
> ----------------------------------------------
>
> Key: HADOOP-206
> URL: https://issues.apache.org/jira/browse/HADOOP-206
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.1.0, 0.1.1, 0.2.0
> Environment: local job tracker
> Reporter: Yoram Arnon
> Assigned To: Runping Qi
> Priority: Minor
> Fix For: 0.14.0
>
>
> when mapred.job.tracker is set to 'local', my setNumReduceTasks call is ignored, and the number of reduce tasks is set at 1.
> This prevents me from locally debugging my partition function, which tries to partition based on the number of reduce tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.