You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Guang Yang <gy...@millennialmedia.com> on 2012/11/13 17:26:05 UTC
Hadoop map-side join
Hi,
I'm trying to use Hadoop map-side join in my application and wondering if anybody knows if there's a way to use it with the new Hadoop API ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for the join is "CompositeInputFormat", which is in the old API package and looks like it expects everything (job configuration, input split, etc) to be from the old API too. This is a problem for me because I'm using the new API to create my map/reduce jobs so I can't just use "CompositeInputFormat" as my job's input format. I wonder if the only way to get the map-side join work is to use the old API to create map/reduce jobs. I appreciate any response regarding this issue.
Thanks,
Guang Yang
Re: Hadoop map-side join
Posted by Guang Yang <gy...@millennialmedia.com>.
Thanks for the answer, Chris. Is there a plan to make the fix available to
1.x in the near future?
- Guang
On 11/14/12 12:42 PM, "Chris Douglas" <cd...@apache.org> wrote:
>See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x
>series) -C
>
>On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com>
>wrote:
>> Hi,
>>
>> I'm trying to use Hadoop map-side join in my application and wondering
>>if
>> anybody knows if there's a way to use it with the new Hadoop API
>> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
>> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for
>>the
>> join is "CompositeInputFormat", which is in the old API package and
>>looks
>> like it expects everything (job configuration, input split, etc) to be
>>from
>> the old API too. This is a problem for me because I'm using the new API
>>to
>> create my map/reduce jobs so I can't just use "CompositeInputFormat" as
>>my
>> job's input format. I wonder if the only way to get the map-side join
>>work
>> is to use the old API to create map/reduce jobs. I appreciate any
>>response
>> regarding this issue.
>>
>> Thanks,
>> Guang Yang
Re: Hadoop map-side join
Posted by Guang Yang <gy...@millennialmedia.com>.
Thanks for the answer, Chris. Is there a plan to make the fix available to
1.x in the near future?
- Guang
On 11/14/12 12:42 PM, "Chris Douglas" <cd...@apache.org> wrote:
>See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x
>series) -C
>
>On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com>
>wrote:
>> Hi,
>>
>> I'm trying to use Hadoop map-side join in my application and wondering
>>if
>> anybody knows if there's a way to use it with the new Hadoop API
>> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
>> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for
>>the
>> join is "CompositeInputFormat", which is in the old API package and
>>looks
>> like it expects everything (job configuration, input split, etc) to be
>>from
>> the old API too. This is a problem for me because I'm using the new API
>>to
>> create my map/reduce jobs so I can't just use "CompositeInputFormat" as
>>my
>> job's input format. I wonder if the only way to get the map-side join
>>work
>> is to use the old API to create map/reduce jobs. I appreciate any
>>response
>> regarding this issue.
>>
>> Thanks,
>> Guang Yang
Re: Hadoop map-side join
Posted by Guang Yang <gy...@millennialmedia.com>.
Thanks for the answer, Chris. Is there a plan to make the fix available to
1.x in the near future?
- Guang
On 11/14/12 12:42 PM, "Chris Douglas" <cd...@apache.org> wrote:
>See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x
>series) -C
>
>On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com>
>wrote:
>> Hi,
>>
>> I'm trying to use Hadoop map-side join in my application and wondering
>>if
>> anybody knows if there's a way to use it with the new Hadoop API
>> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
>> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for
>>the
>> join is "CompositeInputFormat", which is in the old API package and
>>looks
>> like it expects everything (job configuration, input split, etc) to be
>>from
>> the old API too. This is a problem for me because I'm using the new API
>>to
>> create my map/reduce jobs so I can't just use "CompositeInputFormat" as
>>my
>> job's input format. I wonder if the only way to get the map-side join
>>work
>> is to use the old API to create map/reduce jobs. I appreciate any
>>response
>> regarding this issue.
>>
>> Thanks,
>> Guang Yang
Re: Hadoop map-side join
Posted by Guang Yang <gy...@millennialmedia.com>.
Thanks for the answer, Chris. Is there a plan to make the fix available to
1.x in the near future?
- Guang
On 11/14/12 12:42 PM, "Chris Douglas" <cd...@apache.org> wrote:
>See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x
>series) -C
>
>On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com>
>wrote:
>> Hi,
>>
>> I'm trying to use Hadoop map-side join in my application and wondering
>>if
>> anybody knows if there's a way to use it with the new Hadoop API
>> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
>> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for
>>the
>> join is "CompositeInputFormat", which is in the old API package and
>>looks
>> like it expects everything (job configuration, input split, etc) to be
>>from
>> the old API too. This is a problem for me because I'm using the new API
>>to
>> create my map/reduce jobs so I can't just use "CompositeInputFormat" as
>>my
>> job's input format. I wonder if the only way to get the map-side join
>>work
>> is to use the old API to create map/reduce jobs. I appreciate any
>>response
>> regarding this issue.
>>
>> Thanks,
>> Guang Yang
Re: Hadoop map-side join
Posted by Chris Douglas <cd...@apache.org>.
See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x series) -C
On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com> wrote:
> Hi,
>
> I'm trying to use Hadoop map-side join in my application and wondering if
> anybody knows if there's a way to use it with the new Hadoop API
> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for the
> join is "CompositeInputFormat", which is in the old API package and looks
> like it expects everything (job configuration, input split, etc) to be from
> the old API too. This is a problem for me because I'm using the new API to
> create my map/reduce jobs so I can't just use "CompositeInputFormat" as my
> job's input format. I wonder if the only way to get the map-side join work
> is to use the old API to create map/reduce jobs. I appreciate any response
> regarding this issue.
>
> Thanks,
> Guang Yang
Re: Hadoop map-side join
Posted by Chris Douglas <cd...@apache.org>.
See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x series) -C
On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com> wrote:
> Hi,
>
> I'm trying to use Hadoop map-side join in my application and wondering if
> anybody knows if there's a way to use it with the new Hadoop API
> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for the
> join is "CompositeInputFormat", which is in the old API package and looks
> like it expects everything (job configuration, input split, etc) to be from
> the old API too. This is a problem for me because I'm using the new API to
> create my map/reduce jobs so I can't just use "CompositeInputFormat" as my
> job's input format. I wonder if the only way to get the map-side join work
> is to use the old API to create map/reduce jobs. I appreciate any response
> regarding this issue.
>
> Thanks,
> Guang Yang
Re: Hadoop map-side join
Posted by Chris Douglas <cd...@apache.org>.
See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x series) -C
On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com> wrote:
> Hi,
>
> I'm trying to use Hadoop map-side join in my application and wondering if
> anybody knows if there's a way to use it with the new Hadoop API
> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for the
> join is "CompositeInputFormat", which is in the old API package and looks
> like it expects everything (job configuration, input split, etc) to be from
> the old API too. This is a problem for me because I'm using the new API to
> create my map/reduce jobs so I can't just use "CompositeInputFormat" as my
> job's input format. I wonder if the only way to get the map-side join work
> is to use the old API to create map/reduce jobs. I appreciate any response
> regarding this issue.
>
> Thanks,
> Guang Yang
Re: Hadoop map-side join
Posted by Chris Douglas <cd...@apache.org>.
See https://issues.apache.org/jira/browse/MAPREDUCE-355 (not in 1.x series) -C
On Tue, Nov 13, 2012 at 8:26 AM, Guang Yang <gy...@millennialmedia.com> wrote:
> Hi,
>
> I'm trying to use Hadoop map-side join in my application and wondering if
> anybody knows if there's a way to use it with the new Hadoop API
> ("org.apache.hadoop.mapreduce.*") instead of the old Hadoop API
> ("org.apache.hadoop.mapred.*"). The input format I'm trying to use for the
> join is "CompositeInputFormat", which is in the old API package and looks
> like it expects everything (job configuration, input split, etc) to be from
> the old API too. This is a problem for me because I'm using the new API to
> create my map/reduce jobs so I can't just use "CompositeInputFormat" as my
> job's input format. I wonder if the only way to get the map-side join work
> is to use the old API to create map/reduce jobs. I appreciate any response
> regarding this issue.
>
> Thanks,
> Guang Yang