How is pig so much faster than my java MR job?

[Pavan Sudheendra <pa...@gmail.com>]

Hi all,
I have a question which is bugging me for more than a week.
I'm doing some computation across 3 tables in hbase where 1 table is around
25m rows, 2nd table 5m rows and 3rd table 1m rows..

My Java MR job takes a long time to execute.. (in terms of hours) but a pig
script does the same task in under an hour.. A 6 node cluster FYI

Can anybody tell me why Java MR application is slower than a pig script?

-- 
Regards-
Pavan

Re: How is pig so much faster than my java MR job?

[Pavan Sudheendra <pa...@gmail.com>]

Yes.. Caching is set to 1000 .
Number of mappers in pig is 21, don't know about number of reducers..
Probably less than that..

It does many things basically.. The logic consists of the MR job populating
2 tables, one table which has 4m rows and 1 table which has 50,000 rows..


On Tue, Sep 3, 2013 at 12:33 PM, Anoop John <an...@gmail.com> wrote:

> You are using Scan caching in ur MR java code?   How many mapper and
> reducers in case of pig?  How is ur Java MR job written..  a bit more on
> its logic pls.
>
> -Anoop-
>
> On Tue, Sep 3, 2013 at 11:33 AM, Pavan Sudheendra <pavan0591@gmail.com
> >wrote:
>
> > Hi all,
> > I'm doing a kind of table join across 3 tables in the MR job ( plus doing
> > some computation).. It took nearly 19 hours to run with 21 mappers and 21
> > reducers.. But with pig it ran in less than 2 hours..
> > We are using HBase both as source and sink.. Is this normal?
> >
> >
> > On Mon, Sep 2, 2013 at 7:20 PM, Adrien Mogenet <adrien.mogenet@gmail.com
> > >wrote:
> >
> > > You should have a kind of debug/explain mode in Pig, and will show you
> > how
> > > it does clever things to optimize its excution path.
> > >
> > >
> > > On Mon, Sep 2, 2013 at 3:45 PM, Dhaval Shah <
> prince_mithibai@yahoo.co.in
> > > >wrote:
> > >
> > > > Java MR code is not optimized/efficiently written while Pig is highly
> > > > optimized? Can you give us more details on what exactly you are
> trying
> > to
> > > > do and how your Java MR code is written, how many MR jobs for Java vs
> > Pig
> > > > and so on
> > > >
> > > > Sent from Yahoo! Mail on Android
> > > >
> > > >
> > >
> > >
> > > --
> > > Adrien Mogenet
> > > http://www.borntosegfault.com
> > >
> >
> >
> >
> > --
> > Regards-
> > Pavan
> >
>



-- 
Regards-
Pavan

Re: How is pig so much faster than my java MR job?

[Anoop John <an...@gmail.com>]

You are using Scan caching in ur MR java code?   How many mapper and
reducers in case of pig?  How is ur Java MR job written..  a bit more on
its logic pls.

-Anoop-

On Tue, Sep 3, 2013 at 11:33 AM, Pavan Sudheendra <pa...@gmail.com>wrote:

> Hi all,
> I'm doing a kind of table join across 3 tables in the MR job ( plus doing
> some computation).. It took nearly 19 hours to run with 21 mappers and 21
> reducers.. But with pig it ran in less than 2 hours..
> We are using HBase both as source and sink.. Is this normal?
>
>
> On Mon, Sep 2, 2013 at 7:20 PM, Adrien Mogenet <adrien.mogenet@gmail.com
> >wrote:
>
> > You should have a kind of debug/explain mode in Pig, and will show you
> how
> > it does clever things to optimize its excution path.
> >
> >
> > On Mon, Sep 2, 2013 at 3:45 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
> > >wrote:
> >
> > > Java MR code is not optimized/efficiently written while Pig is highly
> > > optimized? Can you give us more details on what exactly you are trying
> to
> > > do and how your Java MR code is written, how many MR jobs for Java vs
> Pig
> > > and so on
> > >
> > > Sent from Yahoo! Mail on Android
> > >
> > >
> >
> >
> > --
> > Adrien Mogenet
> > http://www.borntosegfault.com
> >
>
>
>
> --
> Regards-
> Pavan
>

Re: How is pig so much faster than my java MR job?

[Pavan Sudheendra <pa...@gmail.com>]

Hi all,
I'm doing a kind of table join across 3 tables in the MR job ( plus doing
some computation).. It took nearly 19 hours to run with 21 mappers and 21
reducers.. But with pig it ran in less than 2 hours..
We are using HBase both as source and sink.. Is this normal?


On Mon, Sep 2, 2013 at 7:20 PM, Adrien Mogenet <ad...@gmail.com>wrote:

> You should have a kind of debug/explain mode in Pig, and will show you how
> it does clever things to optimize its excution path.
>
>
> On Mon, Sep 2, 2013 at 3:45 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
> >wrote:
>
> > Java MR code is not optimized/efficiently written while Pig is highly
> > optimized? Can you give us more details on what exactly you are trying to
> > do and how your Java MR code is written, how many MR jobs for Java vs Pig
> > and so on
> >
> > Sent from Yahoo! Mail on Android
> >
> >
>
>
> --
> Adrien Mogenet
> http://www.borntosegfault.com
>



-- 
Regards-
Pavan

Re: How is pig so much faster than my java MR job?

[Adrien Mogenet <ad...@gmail.com>]

You should have a kind of debug/explain mode in Pig, and will show you how
it does clever things to optimize its excution path.


On Mon, Sep 2, 2013 at 3:45 PM, Dhaval Shah <pr...@yahoo.co.in>wrote:

> Java MR code is not optimized/efficiently written while Pig is highly
> optimized? Can you give us more details on what exactly you are trying to
> do and how your Java MR code is written, how many MR jobs for Java vs Pig
> and so on
>
> Sent from Yahoo! Mail on Android
>
>


-- 
Adrien Mogenet
http://www.borntosegfault.com

Re: How is pig so much faster than my java MR job?

[Dhaval Shah <pr...@yahoo.co.in>]

Java MR code is not optimized/efficiently written while Pig is highly optimized? Can you give us more details on what exactly you are trying to do and how your Java MR code is written, how many MR jobs for Java vs Pig and so on

Sent from Yahoo! Mail on Android