You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Mapred Learn <ma...@gmail.com> on 2011/10/21 23:59:39 UTC
Streaming jar creates only 1 reducer
Hi,
Does streaming jar create 1 reducer by default ? We have reduce tasks per
task tracker configured to be more than 1 but my job has about 150 mappers
and only 1 reducer:
reducer.py basically just reads the line and prints it.
Why doesn't streaming.jar invokes multiple reducers for this case ?
Thanks,
-JJ
Re: Streaming jar creates only 1 reducer
Posted by Mapred Learn <ma...@gmail.com>.
Hi Arun,
Thanks !
I was thinking streaming jar would do that itself but looks like not.
Sent from my iPhone
On Oct 21, 2011, at 11:46 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> You can also use -numReduceTasks <#reduces> option to streaming.
>
> On Oct 21, 2011, at 10:22 PM, Mapred Learn wrote:
>
>> Thanks Harsh !
>> This is exactly what I thought !
>>
>> And don't know what you mean by cross-post ? I just posted to mapred and HDFS mailing lists ? What's your point about cross-pointing ??
>>
>> Sent from my iPhone
>>
>> On Oct 21, 2011, at 8:57 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>>> Mapred,
>>>
>>> You need to pass -Dmapred.reduce.tasks=N along. Reducers are a per-job configurable number, unlike mappers whose numbers can be determined based on inputs.
>>>
>>> P.s. Please do not cross post questions to multiple lists.
>>>
>>> On 22-Oct-2011, at 4:05 AM, Mapred Learn wrote:
>>>
>>>> Do you know what parameters from conf files ?
>>>>
>>>> Thanks,
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Oct 21, 2011, at 3:32 PM, Nick Jones <da...@gmail.com> wrote:
>>>>
>>>>> FWIW, I usually specify the number of reducers in both streaming and
>>>>> against the Java API. The "default" is what's read from your config
>>>>> files on the submitting node.
>>>>>
>>>>> Nick Jones
>>>>>
>>>>> On Oct 21, 2011, at 5:00 PM, Mapred Learn <ma...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer:
>>>>>>
>>>>>> reducer.py basically just reads the line and prints it.
>>>>>>
>>>>>> Why doesn't streaming.jar invokes multiple reducers for this case ?
>>>>>>
>>>>>> Thanks,
>>>>>> -JJ
>>>>>>
>>>>>>
>>>
>
Re: Streaming jar creates only 1 reducer
Posted by Arun C Murthy <ac...@hortonworks.com>.
You can also use -numReduceTasks <#reduces> option to streaming.
On Oct 21, 2011, at 10:22 PM, Mapred Learn wrote:
> Thanks Harsh !
> This is exactly what I thought !
>
> And don't know what you mean by cross-post ? I just posted to mapred and HDFS mailing lists ? What's your point about cross-pointing ??
>
> Sent from my iPhone
>
> On Oct 21, 2011, at 8:57 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Mapred,
>>
>> You need to pass -Dmapred.reduce.tasks=N along. Reducers are a per-job configurable number, unlike mappers whose numbers can be determined based on inputs.
>>
>> P.s. Please do not cross post questions to multiple lists.
>>
>> On 22-Oct-2011, at 4:05 AM, Mapred Learn wrote:
>>
>>> Do you know what parameters from conf files ?
>>>
>>> Thanks,
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 21, 2011, at 3:32 PM, Nick Jones <da...@gmail.com> wrote:
>>>
>>>> FWIW, I usually specify the number of reducers in both streaming and
>>>> against the Java API. The "default" is what's read from your config
>>>> files on the submitting node.
>>>>
>>>> Nick Jones
>>>>
>>>> On Oct 21, 2011, at 5:00 PM, Mapred Learn <ma...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer:
>>>>>
>>>>> reducer.py basically just reads the line and prints it.
>>>>>
>>>>> Why doesn't streaming.jar invokes multiple reducers for this case ?
>>>>>
>>>>> Thanks,
>>>>> -JJ
>>>>>
>>>>>
>>
Re: Streaming jar creates only 1 reducer
Posted by Harsh J <ha...@cloudera.com>.
Mapred,
Glad to know you are able to control your # of reducers now.
I believe you answered yourself there :)
Doing http://en.wikipedia.org/wiki/Crossposting often leads to nose,
confusion and duplicated response efforts on the lists.
On Sat, Oct 22, 2011 at 10:52 AM, Mapred Learn <ma...@gmail.com> wrote:
> Thanks Harsh !
> This is exactly what I thought !
>
> And don't know what you mean by cross-post ? I just posted to mapred and HDFS mailing lists ? What's your point about cross-pointing ??
>
> Sent from my iPhone
>
> On Oct 21, 2011, at 8:57 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Mapred,
>>
>> You need to pass -Dmapred.reduce.tasks=N along. Reducers are a per-job configurable number, unlike mappers whose numbers can be determined based on inputs.
>>
>> P.s. Please do not cross post questions to multiple lists.
>>
>> On 22-Oct-2011, at 4:05 AM, Mapred Learn wrote:
>>
>>> Do you know what parameters from conf files ?
>>>
>>> Thanks,
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 21, 2011, at 3:32 PM, Nick Jones <da...@gmail.com> wrote:
>>>
>>>> FWIW, I usually specify the number of reducers in both streaming and
>>>> against the Java API. The "default" is what's read from your config
>>>> files on the submitting node.
>>>>
>>>> Nick Jones
>>>>
>>>> On Oct 21, 2011, at 5:00 PM, Mapred Learn <ma...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer:
>>>>>
>>>>> reducer.py basically just reads the line and prints it.
>>>>>
>>>>> Why doesn't streaming.jar invokes multiple reducers for this case ?
>>>>>
>>>>> Thanks,
>>>>> -JJ
>>>>>
>>>>>
>>
>
--
Harsh J
Re: Streaming jar creates only 1 reducer
Posted by Mapred Learn <ma...@gmail.com>.
Thanks Harsh !
This is exactly what I thought !
And don't know what you mean by cross-post ? I just posted to mapred and HDFS mailing lists ? What's your point about cross-pointing ??
Sent from my iPhone
On Oct 21, 2011, at 8:57 PM, Harsh J <ha...@cloudera.com> wrote:
> Mapred,
>
> You need to pass -Dmapred.reduce.tasks=N along. Reducers are a per-job configurable number, unlike mappers whose numbers can be determined based on inputs.
>
> P.s. Please do not cross post questions to multiple lists.
>
> On 22-Oct-2011, at 4:05 AM, Mapred Learn wrote:
>
>> Do you know what parameters from conf files ?
>>
>> Thanks,
>>
>> Sent from my iPhone
>>
>> On Oct 21, 2011, at 3:32 PM, Nick Jones <da...@gmail.com> wrote:
>>
>>> FWIW, I usually specify the number of reducers in both streaming and
>>> against the Java API. The "default" is what's read from your config
>>> files on the submitting node.
>>>
>>> Nick Jones
>>>
>>> On Oct 21, 2011, at 5:00 PM, Mapred Learn <ma...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer:
>>>>
>>>> reducer.py basically just reads the line and prints it.
>>>>
>>>> Why doesn't streaming.jar invokes multiple reducers for this case ?
>>>>
>>>> Thanks,
>>>> -JJ
>>>>
>>>>
>
Re: Streaming jar creates only 1 reducer
Posted by Harsh J <ha...@cloudera.com>.
Mapred,
You need to pass -Dmapred.reduce.tasks=N along. Reducers are a per-job configurable number, unlike mappers whose numbers can be determined based on inputs.
P.s. Please do not cross post questions to multiple lists.
On 22-Oct-2011, at 4:05 AM, Mapred Learn wrote:
> Do you know what parameters from conf files ?
>
> Thanks,
>
> Sent from my iPhone
>
> On Oct 21, 2011, at 3:32 PM, Nick Jones <da...@gmail.com> wrote:
>
>> FWIW, I usually specify the number of reducers in both streaming and
>> against the Java API. The "default" is what's read from your config
>> files on the submitting node.
>>
>> Nick Jones
>>
>> On Oct 21, 2011, at 5:00 PM, Mapred Learn <ma...@gmail.com> wrote:
>>
>>> Hi,
>>> Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer:
>>>
>>> reducer.py basically just reads the line and prints it.
>>>
>>> Why doesn't streaming.jar invokes multiple reducers for this case ?
>>>
>>> Thanks,
>>> -JJ
>>>
>>>
Re: Streaming jar creates only 1 reducer
Posted by Mapred Learn <ma...@gmail.com>.
Do you know what parameters from conf files ?
Thanks,
Sent from my iPhone
On Oct 21, 2011, at 3:32 PM, Nick Jones <da...@gmail.com> wrote:
> FWIW, I usually specify the number of reducers in both streaming and
> against the Java API. The "default" is what's read from your config
> files on the submitting node.
>
> Nick Jones
>
> On Oct 21, 2011, at 5:00 PM, Mapred Learn <ma...@gmail.com> wrote:
>
>> Hi,
>> Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer:
>>
>> reducer.py basically just reads the line and prints it.
>>
>> Why doesn't streaming.jar invokes multiple reducers for this case ?
>>
>> Thanks,
>> -JJ
>>
>>
Re: Streaming jar creates only 1 reducer
Posted by Nick Jones <da...@gmail.com>.
FWIW, I usually specify the number of reducers in both streaming and
against the Java API. The "default" is what's read from your config
files on the submitting node.
Nick Jones
On Oct 21, 2011, at 5:00 PM, Mapred Learn <ma...@gmail.com> wrote:
> Hi,
> Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer:
>
> reducer.py basically just reads the line and prints it.
>
> Why doesn't streaming.jar invokes multiple reducers for this case ?
>
> Thanks,
> -JJ
>
>