You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2014/01/08 08:32:30 UTC
issue about how to assiging map output to reducer?
hi,maillist:
i look the containers log from " hadoop fs -cat
/var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853"
and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7
to fetcher 4 and assiging 11 to fetcher 3,my question is why not
" assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to
fetcher 3 " ?
2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion
Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher:
attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs
2014-01-08 11:28:00,348 INFO [fetcher#5]
org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
CHBM223:8080 with 7 to fetcher#5
2014-01-08 11:28:00,349 INFO [fetcher#5]
org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
to CHBM223:8080 to fetcher#5
2014-01-08 11:28:00,349 INFO [fetcher#4]
org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
CHBM222:8080 with 7 to fetcher#4
2014-01-08 11:28:00,349 INFO [fetcher#4]
org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
to CHBM222:8080 to fetcher#4
2014-01-08 11:28:00,352 INFO [fetcher#3]
org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
CHBM221:8080 with 11 to fetcher#3
2014-01-08 11:28:00,352 INFO [fetcher#3]
org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11
to CHBM221:8080 to fetcher#3
Re: issue about how to assiging map output to reducer?
Posted by Chris Mawata <ch...@gmail.com>.
Depends on the distribution of the keys and how the partitioner is
assigning keys to reducers. (Remember that pairs with the same key have to
go to the same reducer).
Chris
On Jan 8, 2014 2:33 AM, "ch huang" <ju...@gmail.com> wrote:
> hi,maillist:
> i look the containers log from " hadoop fs -cat
> /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853"
>
> and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7
> to fetcher 4 and assiging 11 to fetcher 3,my question is why not
> " assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to
> fetcher 3 " ?
>
> 2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion
> Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher:
> attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs
> 2014-01-08 11:28:00,348 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM223:8080 with 7 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM223:8080 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM222:8080 with 7 to fetcher#4
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM222:8080 to fetcher#4
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM221:8080 with 11 to fetcher#3
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11
> to CHBM221:8080 to fetcher#3
>
Re: issue about how to assiging map output to reducer?
Posted by Chris Mawata <ch...@gmail.com>.
Depends on the distribution of the keys and how the partitioner is
assigning keys to reducers. (Remember that pairs with the same key have to
go to the same reducer).
Chris
On Jan 8, 2014 2:33 AM, "ch huang" <ju...@gmail.com> wrote:
> hi,maillist:
> i look the containers log from " hadoop fs -cat
> /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853"
>
> and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7
> to fetcher 4 and assiging 11 to fetcher 3,my question is why not
> " assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to
> fetcher 3 " ?
>
> 2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion
> Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher:
> attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs
> 2014-01-08 11:28:00,348 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM223:8080 with 7 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM223:8080 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM222:8080 with 7 to fetcher#4
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM222:8080 to fetcher#4
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM221:8080 with 11 to fetcher#3
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11
> to CHBM221:8080 to fetcher#3
>
Re: issue about how to assiging map output to reducer?
Posted by Chris Mawata <ch...@gmail.com>.
Depends on the distribution of the keys and how the partitioner is
assigning keys to reducers. (Remember that pairs with the same key have to
go to the same reducer).
Chris
On Jan 8, 2014 2:33 AM, "ch huang" <ju...@gmail.com> wrote:
> hi,maillist:
> i look the containers log from " hadoop fs -cat
> /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853"
>
> and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7
> to fetcher 4 and assiging 11 to fetcher 3,my question is why not
> " assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to
> fetcher 3 " ?
>
> 2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion
> Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher:
> attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs
> 2014-01-08 11:28:00,348 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM223:8080 with 7 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM223:8080 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM222:8080 with 7 to fetcher#4
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM222:8080 to fetcher#4
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM221:8080 with 11 to fetcher#3
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11
> to CHBM221:8080 to fetcher#3
>
Re: issue about how to assiging map output to reducer?
Posted by Chris Mawata <ch...@gmail.com>.
Depends on the distribution of the keys and how the partitioner is
assigning keys to reducers. (Remember that pairs with the same key have to
go to the same reducer).
Chris
On Jan 8, 2014 2:33 AM, "ch huang" <ju...@gmail.com> wrote:
> hi,maillist:
> i look the containers log from " hadoop fs -cat
> /var/log/hadoop-yarn/apps/root/logs/application_1388730279827_2770/CHBM221_50853"
>
> and log say it get 25 map output , and assiging 7 to fetcher 5, assiging 7
> to fetcher 4 and assiging 11 to fetcher 3,my question is why not
> " assiging 8 to fetcher 5, assiging 8 to fetcher 4 and assiging 9 to
> fetcher 3 " ?
>
> 2014-01-08 11:28:00,346 INFO [EventFetcher for fetching Map Completion
> Events] org.apache.hadoop.mapreduce.task.reduce.EventFetcher:
> attempt_1388730279827_2770_r_000000_0: Got 25 new map-outputs
> 2014-01-08 11:28:00,348 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM223:8080 with 7 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#5]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM223:8080 to fetcher#5
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM222:8080 with 7 to fetcher#4
> 2014-01-08 11:28:00,349 INFO [fetcher#4]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 7 of 7
> to CHBM222:8080 to fetcher#4
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: Assiging
> CHBM221:8080 with 11 to fetcher#3
> 2014-01-08 11:28:00,352 INFO [fetcher#3]
> org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler: assigned 11 of 11
> to CHBM221:8080 to fetcher#3
>