You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by samir das mohapatra <sa...@gmail.com> on 2012/05/23 21:47:27 UTC
Right way to implement MR ?
Hi All,
How to compare to input file In M/R Job.
let A Log file around 30GB
and B Log file size is around 60 GB
I wanted to know how i will define <K,V> inside the mapper.
Thanks
samir.
Re: Right way to implement MR ?
Posted by Arun C Murthy <ac...@hortonworks.com>.
You might want to start with http://hadoop.apache.org/common/docs/stable/mapred_tutorial.html.
Arun
On May 23, 2012, at 12:47 PM, samir das mohapatra wrote:
> Hi All,
> How to compare to input file In M/R Job.
> let A Log file around 30GB
> and B Log file size is around 60 GB
>
> I wanted to know how i will define <K,V> inside the mapper.
>
> Thanks
> samir.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Right way to implement MR ?
Posted by samir das mohapatra <sa...@gmail.com>.
Thanks
Harsh J for your help.
On Thu, May 24, 2012 at 1:24 AM, Harsh J <ha...@cloudera.com> wrote:
> Samir,
>
> You can use MultipleInputs for multiple forms of inputs per mapper
> (with their own input K/V types, but common output K/V types) with a
> common reduce-side join/compare.
>
> See
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/MultipleInputs.html
> .
>
> On Thu, May 24, 2012 at 1:17 AM, samir das mohapatra
> <sa...@gmail.com> wrote:
> > Hi All,
> > How to compare to input file In M/R Job.
> > let A Log file around 30GB
> > and B Log file size is around 60 GB
> >
> > I wanted to know how i will define <K,V> inside the mapper.
> >
> > Thanks
> > samir.
>
>
>
> --
> Harsh J
>
Re: Right way to implement MR ?
Posted by Harsh J <ha...@cloudera.com>.
Samir,
You can use MultipleInputs for multiple forms of inputs per mapper
(with their own input K/V types, but common output K/V types) with a
common reduce-side join/compare.
See http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/MultipleInputs.html.
On Thu, May 24, 2012 at 1:17 AM, samir das mohapatra
<sa...@gmail.com> wrote:
> Hi All,
> How to compare to input file In M/R Job.
> let A Log file around 30GB
> and B Log file size is around 60 GB
>
> I wanted to know how i will define <K,V> inside the mapper.
>
> Thanks
> samir.
--
Harsh J
RE: Right way to implement MR ?
Posted by Shailesh Dargude <Sh...@symantec.com>.
Samir,
Depends upon your data format and what you want to achieve. More information could have helped....
-Shailesh.
-----Original Message-----
From: samir das mohapatra [mailto:samir.helpdoc@gmail.com]
Sent: Thursday, May 24, 2012 1:17 AM
To: common-user@hadoop.apache.org
Subject: Right way to implement MR ?
Hi All,
How to compare to input file In M/R Job.
let A Log file around 30GB
and B Log file size is around 60 GB
I wanted to know how i will define <K,V> inside the mapper.
Thanks
samir.