You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Henning Blohm <he...@zfabrik.de> on 2010/11/09 16:12:57 UTC
new mapreduce API and NLineInputFormat
I am using the new mapreduce.* API in my jobs on hadoop 0.20.2. - I
actually
have some utilities around job scheduling and such.
Now I would like to use NLineInputFormat to parallelize some data import
task into a number of mappers.
However in 0.20.2 you cannot call
job.setInputFormatClass(NLineInputFormat.class);
as NLineInputFormat does not extend the "right" InputFormat interface
(in
contrast to the 0.21 version).
Is there any way to make this work on 0.20.2?
Thanks,
Henning
Re: new mapreduce API and NLineInputFormat
Posted by Edward Capriolo <ed...@gmail.com>.
On Fri, Jan 14, 2011 at 5:05 PM, Attila Csordas <at...@gmail.com> wrote:
> Hi,
>
> what other jars should be added to the build path from 0.21.0
> besides hadoop-common-0.21.0.jar in order to make 0.21.0 NLineInputFormat
> work in 0.20.2 as suggested below?
>
> Generally can somebody provide me a working example code?
>
> Thanks,
> Attila
>
>
> On Wed, Nov 10, 2010 at 5:06 AM, Harsh J <qw...@gmail.com> wrote:
>
>> Hi,
>>
>> On Tue, Nov 9, 2010 at 8:42 PM, Henning Blohm <he...@zfabrik.de>
>> wrote:
>> >
>> > However in 0.20.2 you cannot call
>> >
>> > job.setInputFormatClass(NLineInputFormat.class);
>> >
>> > as NLineInputFormat does not extend the "right" InputFormat interface
>> > (in
>> > contrast to the 0.21 version).
>> >
>>
>> 0.20.2 does not have it. You can pull the implementation from 0.21.0
>> and use it from within your packages if you require it, though. There
>> should be no problems in doing it.
>>
>> Here's the file from the 0.21 branch:
>>
>> http://svn.apache.org/viewvc/hadoop/mapreduce/tags/release-0.21.0/src/java/org/apache/hadoop/mapreduce/lib/input/NLineInputFormat.java?view=co
>>
>> --
>> Harsh J
>> www.harshj.com
>>
>
Should should not add 0.21 jars to the 0.20 classpath. It will likely
cause a conflict. Just put NLineInputFormat.java in your own project
and try to get it to compile.
Edward
Re: new mapreduce API and NLineInputFormat
Posted by Attila Csordas <at...@gmail.com>.
Hi,
what other jars should be added to the build path from 0.21.0
besides hadoop-common-0.21.0.jar in order to make 0.21.0 NLineInputFormat
work in 0.20.2 as suggested below?
Generally can somebody provide me a working example code?
Thanks,
Attila
On Wed, Nov 10, 2010 at 5:06 AM, Harsh J <qw...@gmail.com> wrote:
> Hi,
>
> On Tue, Nov 9, 2010 at 8:42 PM, Henning Blohm <he...@zfabrik.de>
> wrote:
> >
> > However in 0.20.2 you cannot call
> >
> > job.setInputFormatClass(NLineInputFormat.class);
> >
> > as NLineInputFormat does not extend the "right" InputFormat interface
> > (in
> > contrast to the 0.21 version).
> >
>
> 0.20.2 does not have it. You can pull the implementation from 0.21.0
> and use it from within your packages if you require it, though. There
> should be no problems in doing it.
>
> Here's the file from the 0.21 branch:
>
> http://svn.apache.org/viewvc/hadoop/mapreduce/tags/release-0.21.0/src/java/org/apache/hadoop/mapreduce/lib/input/NLineInputFormat.java?view=co
>
> --
> Harsh J
> www.harshj.com
>
Re: new mapreduce API and NLineInputFormat
Posted by Henning Blohm <he...@zfabrik.de>.
Right, that's what I will do.
Thanks,
Henning
On Wed, 2010-11-10 at 10:36 +0530, Harsh J wrote:
> Hi,
>
> On Tue, Nov 9, 2010 at 8:42 PM, Henning Blohm <he...@zfabrik.de> wrote:
> >
> > However in 0.20.2 you cannot call
> >
> > job.setInputFormatClass(NLineInputFormat.class);
> >
> > as NLineInputFormat does not extend the "right" InputFormat interface
> > (in
> > contrast to the 0.21 version).
> >
>
> 0.20.2 does not have it. You can pull the implementation from 0.21.0
> and use it from within your packages if you require it, though. There
> should be no problems in doing it.
>
> Here's the file from the 0.21 branch:
> http://svn.apache.org/viewvc/hadoop/mapreduce/tags/release-0.21.0/src/java/org/apache/hadoop/mapreduce/lib/input/NLineInputFormat.java?view=co
>
Re: new mapreduce API and NLineInputFormat
Posted by Harsh J <qw...@gmail.com>.
Hi,
On Tue, Nov 9, 2010 at 8:42 PM, Henning Blohm <he...@zfabrik.de> wrote:
>
> However in 0.20.2 you cannot call
>
> job.setInputFormatClass(NLineInputFormat.class);
>
> as NLineInputFormat does not extend the "right" InputFormat interface
> (in
> contrast to the 0.21 version).
>
0.20.2 does not have it. You can pull the implementation from 0.21.0
and use it from within your packages if you require it, though. There
should be no problems in doing it.
Here's the file from the 0.21 branch:
http://svn.apache.org/viewvc/hadoop/mapreduce/tags/release-0.21.0/src/java/org/apache/hadoop/mapreduce/lib/input/NLineInputFormat.java?view=co
--
Harsh J
www.harshj.com