You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Matthew John <tm...@gmail.com> on 2010/10/14 16:03:02 UTC

Multiple Input Data Processing using MapReduce

Hi all ,

 I have been recently working on a task where I need to take in two input
(types)  files , compare them and produce a result from it using a logic.
But as I understand simple MapReduce implementations are for processing a
single input type. The closest implementation I could think of similar to my
work is Join MapReduce. But I am not able to understand much from the
example provided in Hadoop .. Can someone provide a good pointer to such
multiple input data processing ( or Join ) in mapreduce . It will also be
great if you can send in some sample code for the same.

Thanks ,

Matthew

Re: Multiple Input Data Processing using MapReduce

Posted by Michael Toback <mi...@gmail.com>.
I am faced with a similar problem.

I want to process an entire set of bugs including their entire history.
Once. Then, incrementally process a combination of the latest output + the
changes since last processed.

I hit upon a way of handling multiple outputs. Perhaps if there was
something in the data format that told you where the data came from, the
same mapper could process both and the reducer could merge them?

Michael Toback
SMTS, VMWare
-- 
View this message in context: http://lucene.472066.n3.nabble.com/Multiple-Input-Data-Processing-using-MapReduce-tp1701199p2165027.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Re: Multiple Input Data Processing using MapReduce

Posted by Shi Yu <sh...@uchicago.edu>.
Hi Matthew,

I have a same problem here (see 
http://www.listware.net/201009/hadoop-common-user/81228-return-a-parameter-using-map-only.html). 
I was planning to use join mapper (or mapper chain) to handle two 
different inputs. The problem was the mapper seems cannot return value 
directly to each other. Then I have to find out the best settings in 
heap table, MongoDB, memcached, TokyoCabinet, MapFile, etc. etc.  to let 
the multiple mappers talk efficiently.

Shi

On 2010-10-14 9:03, Matthew John wrote:
> Hi all ,
>
>   I have been recently working on a task where I need to take in two input
> (types)  files , compare them and produce a result from it using a logic.
> But as I understand simple MapReduce implementations are for processing a
> single input type. The closest implementation I could think of similar to my
> work is Join MapReduce. But I am not able to understand much from the
> example provided in Hadoop .. Can someone provide a good pointer to such
> multiple input data processing ( or Join ) in mapreduce . It will also be
> great if you can send in some sample code for the same.
>
> Thanks ,
>
> Matthew
>
>    


-- 
Postdoctoral Scholar
Institute for Genomics and Systems Biology
Department of Medicine, the University of Chicago
Knapp Center for Biomedical Discovery
900 E. 57th St. Room 10148
Chicago, IL 60637, US
Tel: 773-702-6799


Re: Multiple Input Data Processing using MapReduce

Posted by Harsh J <qw...@gmail.com>.
It is map.input.file [.start and .length also relate to the InputSplit
for the mapper]
For more: http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#Task+JVM+Reuse

With a custom RR, you can put in this value yourself
(FileSplit.getPath()) before control heads to the Mapper/MapRunner.

On Fri, Dec 31, 2010 at 6:17 PM, Zhou, Yunqing <az...@gmail.com> wrote:
> You can use "map.input.split"(something like that, I can't remember..) param
> in Configuration.
> this param contains the input file path, you can use it to branch your logic
> this param can be found in TextInputFormat.java

-- 
Harsh J
www.harshj.com

Re: Multiple Input Data Processing using MapReduce

Posted by "Zhou, Yunqing" <az...@gmail.com>.
You can use "map.input.split"(something like that, I can't remember..) param
in Configuration.
this param contains the input file path, you can use it to branch your logic
this param can be found in TextInputFormat.java

On Thu, Oct 14, 2010 at 10:03 PM, Matthew John
<tm...@gmail.com>wrote:

> Hi all ,
>
>  I have been recently working on a task where I need to take in two input
> (types)  files , compare them and produce a result from it using a logic.
> But as I understand simple MapReduce implementations are for processing a
> single input type. The closest implementation I could think of similar to
> my
> work is Join MapReduce. But I am not able to understand much from the
> example provided in Hadoop .. Can someone provide a good pointer to such
> multiple input data processing ( or Join ) in mapreduce . It will also be
> great if you can send in some sample code for the same.
>
> Thanks ,
>
> Matthew
>