You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jim Donofrio <do...@gmail.com> on 2012/05/05 17:50:08 UTC
cannot use a map side join to merge the output of multiple map side
joins
I am trying to use a map side join to merge the output of multiple map
side joins. This is failing because of the below code in
JobClient.writeOldSplits which reorders the splits from largest to
smallest. Why is that done, is that so that the largest split which will
take the longest gets processed first?
Each map side join then fails to name its part-* files with the same
number as the incoming partition so files that named part-00000 that go
into the first map side join get outputted to part-00010 while another
one of the first level map side joins sends files named part-00000 to
part-00005. The second level map side join then does not get the input
splits in partitioner order from each first level map side join output
directory.
I can think of only 2 fixes, add some conf property to allow turning off
the below sorting OR extend FileOutputCommitter to rename the outputs of
the first level map side join to merge_part-the orginal partition
number. Any other solutions?
// sort the splits into order based on size, so that the biggest
// go first
Arrays.sort(splits, new
Comparator<org.apache.hadoop.mapred.InputSplit>() {
public int compare(org.apache.hadoop.mapred.InputSplit a,
org.apache.hadoop.mapred.InputSplit b) {
try {
long left = a.getLength();
long right = b.getLength();
if (left == right) {
return 0;
} else if (left < right) {
return 1;
} else {
return -1;
}
Re: cannot use a map side join to merge the output of multiple map
side joins
Posted by JunYong Li <li...@gmail.com>.
could you complain the problem more clear?
2012/5/5 Jim Donofrio <do...@gmail.com>
> I am trying to use a map side join to merge the output of multiple map
> side joins. This is failing because of the below code in
> JobClient.writeOldSplits which reorders the splits from largest to
> smallest. Why is that done, is that so that the largest split which will
> take the longest gets processed first?
>
> Each map side join then fails to name its part-* files with the same
> number as the incoming partition so files that named part-00000 that go
> into the first map side join get outputted to part-00010 while another one
> of the first level map side joins sends files named part-00000 to
> part-00005. The second level map side join then does not get the input
> splits in partitioner order from each first level map side join output
> directory.
>
> I can think of only 2 fixes, add some conf property to allow turning off
> the below sorting OR extend FileOutputCommitter to rename the outputs of
> the first level map side join to merge_part-the orginal partition number.
> Any other solutions?
>
> // sort the splits into order based on size, so that the biggest
> // go first
> Arrays.sort(splits, new Comparator<org.apache.hadoop.**mapred.InputSplit>()
> {
> public int compare(org.apache.hadoop.**mapred.InputSplit a,
> org.apache.hadoop.mapred.**InputSplit b) {
> try {
> long left = a.getLength();
> long right = b.getLength();
> if (left == right) {
> return 0;
> } else if (left < right) {
> return 1;
> } else {
> return -1;
> }
>
--
Regards
Junyong
Re: cannot use a map side join to merge the output of multiple map
side joins
Posted by Jim Donofrio <do...@gmail.com>.
I ended up just using a MultiNamedMultipleOutput with the dynamic part
of the multioutput set to the partition number from one of the
filesplit's inside of the CompositeInputSplit
On 05/07/2012 11:19 AM, Robert Evans wrote:
> I believe that you are correct about the split processing. It orders the splits by size so that the largest splits are processed first. This allows for the smaller splits to potentially fill in the gaps. As far as a fix is concerned I think overriding the file name in the file output committer is a much more straight forward solution to the issue.
>
> --Bobby Evans
>
> On 5/5/12 10:50 AM, "Jim Donofrio"<do...@gmail.com> wrote:
>
> I am trying to use a map side join to merge the output of multiple map
> side joins. This is failing because of the below code in
> JobClient.writeOldSplits which reorders the splits from largest to
> smallest. Why is that done, is that so that the largest split which will
> take the longest gets processed first?
>
> Each map side join then fails to name its part-* files with the same
> number as the incoming partition so files that named part-00000 that go
> into the first map side join get outputted to part-00010 while another
> one of the first level map side joins sends files named part-00000 to
> part-00005. The second level map side join then does not get the input
> splits in partitioner order from each first level map side join output
> directory.
>
> I can think of only 2 fixes, add some conf property to allow turning off
> the below sorting OR extend FileOutputCommitter to rename the outputs of
> the first level map side join to merge_part-the orginal partition
> number. Any other solutions?
>
> // sort the splits into order based on size, so that the biggest
> // go first
> Arrays.sort(splits, new
> Comparator<org.apache.hadoop.mapred.InputSplit>() {
> public int compare(org.apache.hadoop.mapred.InputSplit a,
> org.apache.hadoop.mapred.InputSplit b) {
> try {
> long left = a.getLength();
> long right = b.getLength();
> if (left == right) {
> return 0;
> } else if (left< right) {
> return 1;
> } else {
> return -1;
> }
>
>
Re: cannot use a map side join to merge the output of multiple map
side joins
Posted by Robert Evans <ev...@yahoo-inc.com>.
I believe that you are correct about the split processing. It orders the splits by size so that the largest splits are processed first. This allows for the smaller splits to potentially fill in the gaps. As far as a fix is concerned I think overriding the file name in the file output committer is a much more straight forward solution to the issue.
--Bobby Evans
On 5/5/12 10:50 AM, "Jim Donofrio" <do...@gmail.com> wrote:
I am trying to use a map side join to merge the output of multiple map
side joins. This is failing because of the below code in
JobClient.writeOldSplits which reorders the splits from largest to
smallest. Why is that done, is that so that the largest split which will
take the longest gets processed first?
Each map side join then fails to name its part-* files with the same
number as the incoming partition so files that named part-00000 that go
into the first map side join get outputted to part-00010 while another
one of the first level map side joins sends files named part-00000 to
part-00005. The second level map side join then does not get the input
splits in partitioner order from each first level map side join output
directory.
I can think of only 2 fixes, add some conf property to allow turning off
the below sorting OR extend FileOutputCommitter to rename the outputs of
the first level map side join to merge_part-the orginal partition
number. Any other solutions?
// sort the splits into order based on size, so that the biggest
// go first
Arrays.sort(splits, new
Comparator<org.apache.hadoop.mapred.InputSplit>() {
public int compare(org.apache.hadoop.mapred.InputSplit a,
org.apache.hadoop.mapred.InputSplit b) {
try {
long left = a.getLength();
long right = b.getLength();
if (left == right) {
return 0;
} else if (left < right) {
return 1;
} else {
return -1;
}