Help in Hadoop

[aa...@buffalo.edu]

Hello Everybody,
                I have a doubt in a map reduce program and I would appreciate any
help. I run the program using the command bin/hadoop jar HomeWork.jar prg1 input
output. Ideally from within prg1, I want to sequentially launch 10 map- reduce
tasks. I want to store the output of all these map reduce tasks in some file.
Currently I have kept the input format and output format of the jobs as
TextInputFormat and TextOutputFormat respectively. Now I have the following
questions.

1. When I run more than 1 task from the same program, the output file of all the
tasks is same. The framework does not allows the 2   map reduce task to have the
same output file as task 1.

2. Before the 2 task launches I also get this error 

Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already
initialized

3. When the 2 map reduce tasks writes its output to file "output", wont the
previous content of this file get over written ?

Thank You

Abhishek Agrawal

SUNY- Buffalo
(716-435-7122)

Re: Help in Hadoop

[Edmund Kohlwey <ek...@gmail.com>]

> 1. When I run more than 1 task from the same program, the output file of all the
> tasks is same. The framework does not allows the 2   map reduce task to have the
> same output file as task 1.
>    
That's correct. Your instructor probably intends for you to find a way 
to merge the files. There's a lot of ways you might do this, both 
through a Map/Reduce job or just single-threaded on the child machine. 
The documentation for the FileSystem class and/or SequenceFile.Reader 
and SequenceFile.Writer may help. Also remember that you can control a 
lot of how a job runs by changing settings in the JobConf.
> 3. When the 2 map reduce tasks writes its output to file "output", wont the
> previous content of this file get over written
Actually, this will produce an exception since the file already exists.