You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Dexin Wang <wa...@gmail.com> on 2011/12/19 23:23:43 UTC
pig 0.9 slower in local mode?
I recently switched to pig 0.9.1 and noticed it runs slower than previous
version (like 0.6 which was only recent version supported on Amazon couple
of months ago) in local mode. Haven't tried the timing in hadoop mode yet.
I figure it is probably due to some extra debugging or some parameter.
Anything I can do to make it faster?
Thanks,
Dexin
Re: pig 0.9 slower in local mode?
Posted by Dexin Wang <wa...@gmail.com>.
Cool. Looking forward to trying out 0.9.2+patch.
On Mon, Dec 19, 2011 at 2:31 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Yes, starting with 0.7 local mode was moved to sit on top of Hadoop's
> local mode (instead of being a completely separate implementation).
> This had two effects, one desirable, one not so much:
>
> 1) get rid of surprises / bugs caused by having to support 2
> completely separate implementations of the runtime
> 2) slow the thing down a lot.
>
> Julien sped up local mode quite a bit in 0.9.2, and we have another
> patch coming that further removes unnecessary slowness, but I'm afraid
> hadoop-based local mode will never be quite as fast as the old
> local-mode...
>
> D
>
> On Mon, Dec 19, 2011 at 2:23 PM, Dexin Wang <wa...@gmail.com> wrote:
> > I recently switched to pig 0.9.1 and noticed it runs slower than previous
> > version (like 0.6 which was only recent version supported on Amazon
> couple
> > of months ago) in local mode. Haven't tried the timing in hadoop mode
> yet.
> >
> > I figure it is probably due to some extra debugging or some parameter.
> > Anything I can do to make it faster?
> >
> > Thanks,
> > Dexin
>
Re: pig 0.9 slower in local mode?
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Yes, starting with 0.7 local mode was moved to sit on top of Hadoop's
local mode (instead of being a completely separate implementation).
This had two effects, one desirable, one not so much:
1) get rid of surprises / bugs caused by having to support 2
completely separate implementations of the runtime
2) slow the thing down a lot.
Julien sped up local mode quite a bit in 0.9.2, and we have another
patch coming that further removes unnecessary slowness, but I'm afraid
hadoop-based local mode will never be quite as fast as the old
local-mode...
D
On Mon, Dec 19, 2011 at 2:23 PM, Dexin Wang <wa...@gmail.com> wrote:
> I recently switched to pig 0.9.1 and noticed it runs slower than previous
> version (like 0.6 which was only recent version supported on Amazon couple
> of months ago) in local mode. Haven't tried the timing in hadoop mode yet.
>
> I figure it is probably due to some extra debugging or some parameter.
> Anything I can do to make it faster?
>
> Thanks,
> Dexin
Re: pig 0.9 slower in local mode?
Posted by Koji Noguchi <kn...@yahoo-inc.com>.
https://issues.apache.org/jira/browse/PIG-2128
could be related.
Koji
On 12/19/11 2:23 PM, "Dexin Wang" <wa...@gmail.com> wrote:
> I recently switched to pig 0.9.1 and noticed it runs slower than previous
> version (like 0.6 which was only recent version supported on Amazon couple
> of months ago) in local mode. Haven't tried the timing in hadoop mode yet.
>
> I figure it is probably due to some extra debugging or some parameter.
> Anything I can do to make it faster?
>
> Thanks,
> Dexin