Map Reduce when downloading files...

[Michael Giannakopoulos <mi...@gmail.com>]

Hello guys,
I have written an application that downloads metadata from 3 groups of
Flickr and i implement a map/reduce task so as metadata to be processed by 3
different mappers (each corresponds to one group...). My app runs on single
mode, but when i try to run it in a pseudo-distributed mode hadoop keeps
showing me java.lang.NullPointerException. This happens before even the map
task starts. I suspect that something happens when the mapper tries to
create keys, or something with the encoding of the input file for the
mapper. Exception is the following one:
java.lang.NullPointerException
    at metaFlickrPro.PhotosDownload$MapClass.map(PhotosDownload.java:124)
    at metaFlickrPro.PhotosDownload$MapClass.map(PhotosDownload.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
    at org.apache.hadoop.mapred.Child.main(Child.java:170)

where "metaFlickrPro" is the package of my application and PhotosDownload is
the public class that contains the static classes of mapper and reducer...
Please, if anyone has encountered a similar problem, i would be grateful if
he/she answered. I have attached the log file from the map-reduce process. I
run hadoop 0.20.2 on Ubuntu 10.04...

Re: Map Reduce when downloading files...

[Michael Giannakopoulos <mi...@gmail.com>]

Thanks a lot! I 'll try it...

Re: Map Reduce when downloading files...

[Harsh J <ha...@cloudera.com>]

I haven't gone through the whole thing, but getting the Configuration
object via a static member "conf" set only during submission (main())
will not work - and is probably why there's an NPE.

Use the Context object in the map() call to get a configuration
instance. That is the only right way I know of.

i.e., change 124 to:
groupTask = new GroupTask(context.getConfiguration(), groupID,
PhotosDownload.outputPath, f.getPoolsInterface());

On Wed, May 25, 2011 at 9:37 PM, Michael Giannakopoulos
<mi...@gmail.com> wrote:
> Alright, I 'll send you the code (it's an amateur application). Any help is
> appreciated! (Don't bother with the Flickrj API)... And something else, how
> do you debug a map/reduce app so as to be sure what happens. I use eclipse
> and hadoop's plugin for eclipse (Galileo). Thanks a lot!
>



-- 
Harsh J

Re: Map Reduce when downloading files...

[Michael Giannakopoulos <mi...@gmail.com>]

Alright, I 'll send you the code (it's an amateur application). Any help is
appreciated! (Don't bother with the Flickrj API)... And something else, how
do you debug a map/reduce app so as to be sure what happens. I use eclipse
and hadoop's plugin for eclipse (Galileo). Thanks a lot!

Re: Map Reduce when downloading files...

[Harsh J <ha...@cloudera.com>]

Based on your stacktrace, the 'Task' did begin alright. (This is
post-configuration/setup)

You're getting an NPE on
metaFlickrPro\.PhotosDownload$MapClass\.map(PhotosDownload\.java:124)
Its not possible for us to tell why since the point that was thrown is
from your custom code - and we do not have that.
I can't really tell if this is a framework issue with that trace.

On Wed, May 25, 2011 at 9:21 PM, Michael Giannakopoulos
<mi...@gmail.com> wrote:
> Hello guys,
> I have written an application that downloads metadata from 3 groups of
> Flickr and i implement a map/reduce task so as metadata to be processed by 3
> different mappers (each corresponds to one group...). My app runs on single
> mode, but when i try to run it in a pseudo-distributed mode hadoop keeps
> showing me java.lang.NullPointerException. This happens before even the map
> task starts. I suspect that something happens when the mapper tries to
> create keys, or something with the encoding of the input file for the
> mapper. Exception is the following one:
> java.lang.NullPointerException
>     at metaFlickrPro.PhotosDownload$MapClass.map(PhotosDownload.java:124)
>     at metaFlickrPro.PhotosDownload$MapClass.map(PhotosDownload.java:1)
>     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>     at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
> where "metaFlickrPro" is the package of my application and PhotosDownload is
> the public class that contains the static classes of mapper and reducer...
> Please, if anyone has encountered a similar problem, i would be grateful if
> he/she answered. I have attached the log file from the map-reduce process. I
> run hadoop 0.20.2 on Ubuntu 10.04...
>



-- 
Harsh J