You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Aniket Mokashi <an...@gmail.com> on 2013/10/01 00:37:57 UTC
Re: [Discussion] Any thoughts on PIG-3457?
+1 on reverting PIG-3419 and applying it to tez branch if its blocking
pig-0.12 release.
On Mon, Sep 30, 2013 at 2:42 PM, Cheolsoo Park <pi...@gmail.com> wrote:
> I am waiting for +1 from Twitter.
>
> Like Alan suggested, let's revert PIG-3419 et al in 0.12 first. Then, we
> can decide what to do in trunk. I volunteer to do grunt work since I am the
> one who committed them.
>
>
> On Mon, Sep 30, 2013 at 2:26 PM, Rohini Palaniswamy <
> rohini.aditya@gmail.com
> > wrote:
>
> > +1. I was already asking for keeping the new API changes only in Tez
> branch
> > till it evolves and is finalized, so I have no objections to reverting
> it.
> >
> > Regards,
> > Rohini
> >
> >
> > On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates <ga...@hortonworks.com>
> wrote:
> >
> > > We should separate out two separate concerns. If I understand
> correctly
> > > we don't need any of these changes in 0.12. So we should revert these
> > > patches from the 12 branch so that we can get it released quickly in a
> > > backwards compatible way.
> > >
> > > We will then have plenty of time to discuss the separate question of
> how
> > > we proceed going forward (deprecated APIs or new APIs).
> > >
> > > Alan.
> > >
> > > On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote:
> > >
> > > > Hi Jeremy,
> > > >
> > > > What you're saying makes sense, and patch is welcome. ;-) But
> > complexity
> > > > comes from that there are many classes that are associated with one
> > > > another, and it seems necessary to bring back all of them together in
> > > order
> > > > to provide full backward compatibility.
> > > >
> > > > After spending many hours on the weekend, I concluded that adding
> more
> > > > workarounds (classes, methods, packages, etc) to the current code
> makes
> > > it
> > > > only less maintainable and readable. So I prefer a simpler approach.
> > > >
> > > > For eg, we can just publish two jars - pig.jar w/ old API and
> > pig-new.jar
> > > > w/ new API - maybe not in 0.12 but in 0.13. Since we already have a
> > > > tez-branch, we can use it to manage the new version of classes. Then,
> > > users
> > > > can switch to pig-new.jar gradually in 0.13 and 0.14. When we finally
> > > merge
> > > > tez-branch into trunk, we can publish a single jar again.
> > > >
> > > > Of course, this is not trivial either because we have to maintain two
> > > > branches. But I feel that managing two branches independently is
> easier
> > > > than maintaining all sorts of workarounds for backward compatibility
> in
> > > the
> > > > source code. In addition, we will have more flexibility in terms of
> > > > designing new API because we will be completely free from backward
> > > > compatibility. No?
> > > >
> > > > Thanks,
> > > > Cheolsoo
> > > >
> > > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <jk...@mortardata.com>
> > > wrote:
> > > >
> > > >> What about the option of leaving all of the MR specific logic in the
> > > >> original classes but marking those methods as deprecated and telling
> > > people
> > > >> to switch to using a MR specific object that extends the original
> > class.
> > > >> So for example:
> > > >>
> > > >> JobStats - Reverted to being as it was before PIG-3419 but with all
> MR
> > > >> specific logic deprecated.
> > > >> MRJobStats - Would just extend JobStats.
> > > >>
> > > >> If we did this, external software could switch their code from using
> > > >> JobStats to MRJobStats at their own pace and without breaking
> against
> > > any
> > > >> specific version of Pig. After a few versions the MR specific logic
> > > could
> > > >> be removed from JobStats and pushed into MRJobStats and it shouldn't
> > > break
> > > >> anything for people that had made that change.
> > > >>
> > > >> I'm not familiar with all of the changes in PIG-3419 so this might
> not
> > > work
> > > >> everywhere.
> > > >>
> > > >>
> > > >> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <
> piaozhexiu@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> To be specific, we will need to revert all the following commits in
> > > >> order:
> > > >>>
> > > >>>
> > > >>> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
> > > >>> Author: Cheolsoo Park <ch...@apache.org>
> > > >>> Date: Fri Sep 20 22:47:29 2013 +0000
> > > >>>
> > > >>> PIG-3471: Add a base abstract class for ExecutionEngine
> (cheolsoo)
> > > >>>
> > > >>> git-svn-id:
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
> > > >>>
> > > >>> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
> > > >>> Author: Jianyong Dai <da...@apache.org>
> > > >>> Date: Wed Sep 18 19:09:49 2013 +0000
> > > >>>
> > > >>> PIG-3457: Provide backward compatibility for PigStatsUtil and
> > > >> JobStats
> > > >>>
> > > >>> git-svn-id:
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68
> > > >>>
> > > >>> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7
> > > >>> Author: Cheolsoo Park <ch...@apache.org>
> > > >>> Date: Wed Sep 18 15:37:58 2013 +0000
> > > >>>
> > > >>> PIG-3464: Mark ExecType and ExecutionEngine interfaces as
> evolving
> > > >>> (cheolsoo)
> > > >>>
> > > >>> commit fd8b7cdf9292b305f02386d560c25298ab492a0b
> > > >>> Author: Cheolsoo Park <ch...@apache.org>
> > > >>> Date: Fri Aug 30 20:04:29 2013 +0000
> > > >>>
> > > >>> PIG-3419: Pluggable Execution Engine (achalsoni81 via cheolsoo)
> > > >>>
> > > >>> git-svn-id:
> > > >>>
> > > >>>
> > > >>
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <
> daijy@hortonworks.com>
> > > >>> wrote:
> > > >>>
> > > >>>> Thanks Cheolsoo! My opinion is provide backward compatibility for
> > > >>> PigStats
> > > >>>> is a must, otherwise it could be a havoc. I imagine PigStats is
> > widely
> > > >>> used
> > > >>>> by Pig users via PigRunner and PPNL interface. People use PigStats
> > to
> > > >>>> collect MR job details of the Pig job. Though PigStats is marked
> for
> > > >>>> Evolving, this is mostly for extending PigStats, not limiting
> > PigStats
> > > >> as
> > > >>>> PIG-3419 did. Even if we really need to change, we need to very
> well
> > > >>>> communicate with users over time, Pig 0.12 is not an option.
> > > >>>>
> > > >>>> PIG-3457 is trying to provide a backward compatibility way for
> > > >> PigStats,
> > > >>>> but just like Cheolsoo said, it is far from ideal. I now tend to
> > agree
> > > >>>> Rohini's suggestion on PIG-3419, rollback PIG-3419, until we find
> a
> > > >>> better
> > > >>>> way. Seems PIG-3419 is a little premature. Besides the above
> > mentioned
> > > >>>> PigStats issue, I've already found 2 additional issues:
> > > >>>> 1. "explain" shows unoptimized logical plan instead of optimized
> one
> > > >>>> 2. HangingJobKiller is removed
> > > >>>>
> > > >>>> How does others think?
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Daniel
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park <
> > piaozhexiu@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi devs,
> > > >>>>>
> > > >>>>> PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419> broke
> > > >>> backward
> > > >>>>> compatibility for downstream applications such as Oozie, and
> > > >>>>> PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is
> > > >>>>> trying to fix it.
> > > >>>>>
> > > >>>>> In summary, we need to keep the old MR-specific JobStats and
> > PigStats
> > > >>> for
> > > >>>>> backward compatibility sadly because they're currently exposed in
> > > >>> several
> > > >>>>> user-facing API including PigRunner, PigServer, etc. However,
> this
> > > >>>> defeats
> > > >>>>> the purpose of PIG-3419
> > > >>>>> <https://issues.apache.org/jira/browse/PIG-3419> because
> > > >>>>> we cannot implement non-MR execution engines in the back-end if
> the
> > > >>>>> front-end API is tied to MR.
> > > >>>>>
> > > >>>>> Daniel and I were having a discussion in the jira, but it seems
> > more
> > > >>>>> complicated than I thought. I am wondering whether anyone has a
> > good
> > > >>>>> suggestion on how to solve it. This is blocking the 0.12 release.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Cheolsoo
> > > >>>>>
> > > >>>>
> > > >>>> --
> > > >>>> CONFIDENTIALITY NOTICE
> > > >>>> NOTICE: This message is intended for the use of the individual or
> > > >> entity
> > > >>> to
> > > >>>> which it is addressed and may contain information that is
> > > confidential,
> > > >>>> privileged and exempt from disclosure under applicable law. If the
> > > >> reader
> > > >>>> of this message is not the intended recipient, you are hereby
> > notified
> > > >>> that
> > > >>>> any printing, copying, dissemination, distribution, disclosure or
> > > >>>> forwarding of this communication is strictly prohibited. If you
> have
> > > >>>> received this communication in error, please contact the sender
> > > >>> immediately
> > > >>>> and delete it from your system. Thank You.
> > > >>>>
> > > >>>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Jeremy Karn / Lead Developer
> > > >> MORTAR DATA / 519 277 4391 / www.mortardata.com
> > > >>
> > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
--
"...:::Aniket:::... Quetzalco@tl"
Re: [Discussion] Any thoughts on PIG-3457?
Posted by Daniel Dai <da...@hortonworks.com>.
Thanks a lot!
On Mon, Sep 30, 2013 at 9:28 PM, Cheolsoo Park <pi...@gmail.com> wrote:
> OK, I reverted PIG-3471, 3457, 3464, and 3419 in 0.12:
> http://svn.apache.org/viewvc?view=revision&revision=1527867
>
> You can also view each revert here:
> https://github.com/piaozhexiu/apache-pig/commits/revert
>
> I had to manually resolve some conflicts particularly due to PIG-3430. I
> believe I didn't break anything, but please let me know if I made any
> mistake. Test-commit passes.
>
> Thank you,
> Cheolsoo
>
>
>
> On Mon, Sep 30, 2013 at 3:52 PM, Cheolsoo Park <pi...@gmail.com>
> wrote:
>
> > Thanks Aniket.
> >
> > I'll revert the aforementioned commits in 0.12 tonight. I will leave them
> > in trunk until we decide what to do.
> >
> >
> > On Mon, Sep 30, 2013 at 3:37 PM, Aniket Mokashi <aniket486@gmail.com
> >wrote:
> >
> >> +1 on reverting PIG-3419 and applying it to tez branch if its blocking
> >> pig-0.12 release.
> >>
> >> On Mon, Sep 30, 2013 at 2:42 PM, Cheolsoo Park <pi...@gmail.com>
> >> wrote:
> >>
> >> > I am waiting for +1 from Twitter.
> >> >
> >> > Like Alan suggested, let's revert PIG-3419 et al in 0.12 first. Then,
> we
> >> > can decide what to do in trunk. I volunteer to do grunt work since I
> am
> >> the
> >> > one who committed them.
> >> >
> >> >
> >> > On Mon, Sep 30, 2013 at 2:26 PM, Rohini Palaniswamy <
> >> > rohini.aditya@gmail.com
> >> > > wrote:
> >> >
> >> > > +1. I was already asking for keeping the new API changes only in Tez
> >> > branch
> >> > > till it evolves and is finalized, so I have no objections to
> reverting
> >> > it.
> >> > >
> >> > > Regards,
> >> > > Rohini
> >> > >
> >> > >
> >> > > On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates <ga...@hortonworks.com>
> >> > wrote:
> >> > >
> >> > > > We should separate out two separate concerns. If I understand
> >> > correctly
> >> > > > we don't need any of these changes in 0.12. So we should revert
> >> these
> >> > > > patches from the 12 branch so that we can get it released quickly
> >> in a
> >> > > > backwards compatible way.
> >> > > >
> >> > > > We will then have plenty of time to discuss the separate question
> of
> >> > how
> >> > > > we proceed going forward (deprecated APIs or new APIs).
> >> > > >
> >> > > > Alan.
> >> > > >
> >> > > > On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote:
> >> > > >
> >> > > > > Hi Jeremy,
> >> > > > >
> >> > > > > What you're saying makes sense, and patch is welcome. ;-) But
> >> > > complexity
> >> > > > > comes from that there are many classes that are associated with
> >> one
> >> > > > > another, and it seems necessary to bring back all of them
> >> together in
> >> > > > order
> >> > > > > to provide full backward compatibility.
> >> > > > >
> >> > > > > After spending many hours on the weekend, I concluded that
> adding
> >> > more
> >> > > > > workarounds (classes, methods, packages, etc) to the current
> code
> >> > makes
> >> > > > it
> >> > > > > only less maintainable and readable. So I prefer a simpler
> >> approach.
> >> > > > >
> >> > > > > For eg, we can just publish two jars - pig.jar w/ old API and
> >> > > pig-new.jar
> >> > > > > w/ new API - maybe not in 0.12 but in 0.13. Since we already
> have
> >> a
> >> > > > > tez-branch, we can use it to manage the new version of classes.
> >> Then,
> >> > > > users
> >> > > > > can switch to pig-new.jar gradually in 0.13 and 0.14. When we
> >> finally
> >> > > > merge
> >> > > > > tez-branch into trunk, we can publish a single jar again.
> >> > > > >
> >> > > > > Of course, this is not trivial either because we have to
> maintain
> >> two
> >> > > > > branches. But I feel that managing two branches independently is
> >> > easier
> >> > > > > than maintaining all sorts of workarounds for backward
> >> compatibility
> >> > in
> >> > > > the
> >> > > > > source code. In addition, we will have more flexibility in terms
> >> of
> >> > > > > designing new API because we will be completely free from
> backward
> >> > > > > compatibility. No?
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Cheolsoo
> >> > > > >
> >> > > > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <
> >> jkarn@mortardata.com>
> >> > > > wrote:
> >> > > > >
> >> > > > >> What about the option of leaving all of the MR specific logic
> in
> >> the
> >> > > > >> original classes but marking those methods as deprecated and
> >> telling
> >> > > > people
> >> > > > >> to switch to using a MR specific object that extends the
> original
> >> > > class.
> >> > > > >> So for example:
> >> > > > >>
> >> > > > >> JobStats - Reverted to being as it was before PIG-3419 but with
> >> all
> >> > MR
> >> > > > >> specific logic deprecated.
> >> > > > >> MRJobStats - Would just extend JobStats.
> >> > > > >>
> >> > > > >> If we did this, external software could switch their code from
> >> using
> >> > > > >> JobStats to MRJobStats at their own pace and without breaking
> >> > against
> >> > > > any
> >> > > > >> specific version of Pig. After a few versions the MR specific
> >> logic
> >> > > > could
> >> > > > >> be removed from JobStats and pushed into MRJobStats and it
> >> shouldn't
> >> > > > break
> >> > > > >> anything for people that had made that change.
> >> > > > >>
> >> > > > >> I'm not familiar with all of the changes in PIG-3419 so this
> >> might
> >> > not
> >> > > > work
> >> > > > >> everywhere.
> >> > > > >>
> >> > > > >>
> >> > > > >> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <
> >> > piaozhexiu@gmail.com>
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >>> To be specific, we will need to revert all the following
> >> commits in
> >> > > > >> order:
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
> >> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
> >> > > > >>> Date: Fri Sep 20 22:47:29 2013 +0000
> >> > > > >>>
> >> > > > >>> PIG-3471: Add a base abstract class for ExecutionEngine
> >> > (cheolsoo)
> >> > > > >>>
> >> > > > >>> git-svn-id:
> >> > > > >>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
> >> > > > >>>
> >> > > > >>> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
> >> > > > >>> Author: Jianyong Dai <da...@apache.org>
> >> > > > >>> Date: Wed Sep 18 19:09:49 2013 +0000
> >> > > > >>>
> >> > > > >>> PIG-3457: Provide backward compatibility for PigStatsUtil
> and
> >> > > > >> JobStats
> >> > > > >>>
> >> > > > >>> git-svn-id:
> >> > > > >>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68
> >> > > > >>>
> >> > > > >>> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7
> >> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
> >> > > > >>> Date: Wed Sep 18 15:37:58 2013 +0000
> >> > > > >>>
> >> > > > >>> PIG-3464: Mark ExecType and ExecutionEngine interfaces as
> >> > evolving
> >> > > > >>> (cheolsoo)
> >> > > > >>>
> >> > > > >>> commit fd8b7cdf9292b305f02386d560c25298ab492a0b
> >> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
> >> > > > >>> Date: Fri Aug 30 20:04:29 2013 +0000
> >> > > > >>>
> >> > > > >>> PIG-3419: Pluggable Execution Engine (achalsoni81 via
> >> cheolsoo)
> >> > > > >>>
> >> > > > >>> git-svn-id:
> >> > > > >>>
> >> > > > >>>
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >>
> https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>>
> >> > > > >>> On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <
> >> > daijy@hortonworks.com>
> >> > > > >>> wrote:
> >> > > > >>>
> >> > > > >>>> Thanks Cheolsoo! My opinion is provide backward compatibility
> >> for
> >> > > > >>> PigStats
> >> > > > >>>> is a must, otherwise it could be a havoc. I imagine PigStats
> is
> >> > > widely
> >> > > > >>> used
> >> > > > >>>> by Pig users via PigRunner and PPNL interface. People use
> >> PigStats
> >> > > to
> >> > > > >>>> collect MR job details of the Pig job. Though PigStats is
> >> marked
> >> > for
> >> > > > >>>> Evolving, this is mostly for extending PigStats, not limiting
> >> > > PigStats
> >> > > > >> as
> >> > > > >>>> PIG-3419 did. Even if we really need to change, we need to
> very
> >> > well
> >> > > > >>>> communicate with users over time, Pig 0.12 is not an option.
> >> > > > >>>>
> >> > > > >>>> PIG-3457 is trying to provide a backward compatibility way
> for
> >> > > > >> PigStats,
> >> > > > >>>> but just like Cheolsoo said, it is far from ideal. I now tend
> >> to
> >> > > agree
> >> > > > >>>> Rohini's suggestion on PIG-3419, rollback PIG-3419, until we
> >> find
> >> > a
> >> > > > >>> better
> >> > > > >>>> way. Seems PIG-3419 is a little premature. Besides the above
> >> > > mentioned
> >> > > > >>>> PigStats issue, I've already found 2 additional issues:
> >> > > > >>>> 1. "explain" shows unoptimized logical plan instead of
> >> optimized
> >> > one
> >> > > > >>>> 2. HangingJobKiller is removed
> >> > > > >>>>
> >> > > > >>>> How does others think?
> >> > > > >>>>
> >> > > > >>>> Thanks,
> >> > > > >>>> Daniel
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>>
> >> > > > >>>> On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park <
> >> > > piaozhexiu@gmail.com>
> >> > > > >>>> wrote:
> >> > > > >>>>
> >> > > > >>>>> Hi devs,
> >> > > > >>>>>
> >> > > > >>>>> PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419>
> >> broke
> >> > > > >>> backward
> >> > > > >>>>> compatibility for downstream applications such as Oozie, and
> >> > > > >>>>> PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is
> >> > > > >>>>> trying to fix it.
> >> > > > >>>>>
> >> > > > >>>>> In summary, we need to keep the old MR-specific JobStats and
> >> > > PigStats
> >> > > > >>> for
> >> > > > >>>>> backward compatibility sadly because they're currently
> >> exposed in
> >> > > > >>> several
> >> > > > >>>>> user-facing API including PigRunner, PigServer, etc.
> However,
> >> > this
> >> > > > >>>> defeats
> >> > > > >>>>> the purpose of PIG-3419
> >> > > > >>>>> <https://issues.apache.org/jira/browse/PIG-3419> because
> >> > > > >>>>> we cannot implement non-MR execution engines in the back-end
> >> if
> >> > the
> >> > > > >>>>> front-end API is tied to MR.
> >> > > > >>>>>
> >> > > > >>>>> Daniel and I were having a discussion in the jira, but it
> >> seems
> >> > > more
> >> > > > >>>>> complicated than I thought. I am wondering whether anyone
> has
> >> a
> >> > > good
> >> > > > >>>>> suggestion on how to solve it. This is blocking the 0.12
> >> release.
> >> > > > >>>>>
> >> > > > >>>>> Thanks,
> >> > > > >>>>> Cheolsoo
> >> > > > >>>>>
> >> > > > >>>>
> >> > > > >>>> --
> >> > > > >>>> CONFIDENTIALITY NOTICE
> >> > > > >>>> NOTICE: This message is intended for the use of the
> individual
> >> or
> >> > > > >> entity
> >> > > > >>> to
> >> > > > >>>> which it is addressed and may contain information that is
> >> > > > confidential,
> >> > > > >>>> privileged and exempt from disclosure under applicable law.
> If
> >> the
> >> > > > >> reader
> >> > > > >>>> of this message is not the intended recipient, you are hereby
> >> > > notified
> >> > > > >>> that
> >> > > > >>>> any printing, copying, dissemination, distribution,
> disclosure
> >> or
> >> > > > >>>> forwarding of this communication is strictly prohibited. If
> you
> >> > have
> >> > > > >>>> received this communication in error, please contact the
> sender
> >> > > > >>> immediately
> >> > > > >>>> and delete it from your system. Thank You.
> >> > > > >>>>
> >> > > > >>>
> >> > > > >>
> >> > > > >>
> >> > > > >>
> >> > > > >> --
> >> > > > >>
> >> > > > >> Jeremy Karn / Lead Developer
> >> > > > >> MORTAR DATA / 519 277 4391 / www.mortardata.com
> >> > > > >>
> >> > > >
> >> > > >
> >> > > > --
> >> > > > CONFIDENTIALITY NOTICE
> >> > > > NOTICE: This message is intended for the use of the individual or
> >> > entity
> >> > > to
> >> > > > which it is addressed and may contain information that is
> >> confidential,
> >> > > > privileged and exempt from disclosure under applicable law. If the
> >> > reader
> >> > > > of this message is not the intended recipient, you are hereby
> >> notified
> >> > > that
> >> > > > any printing, copying, dissemination, distribution, disclosure or
> >> > > > forwarding of this communication is strictly prohibited. If you
> have
> >> > > > received this communication in error, please contact the sender
> >> > > immediately
> >> > > > and delete it from your system. Thank You.
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> "...:::Aniket:::... Quetzalco@tl"
> >>
> >
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: [Discussion] Any thoughts on PIG-3457?
Posted by Cheolsoo Park <pi...@gmail.com>.
OK, I reverted PIG-3471, 3457, 3464, and 3419 in 0.12:
http://svn.apache.org/viewvc?view=revision&revision=1527867
You can also view each revert here:
https://github.com/piaozhexiu/apache-pig/commits/revert
I had to manually resolve some conflicts particularly due to PIG-3430. I
believe I didn't break anything, but please let me know if I made any
mistake. Test-commit passes.
Thank you,
Cheolsoo
On Mon, Sep 30, 2013 at 3:52 PM, Cheolsoo Park <pi...@gmail.com> wrote:
> Thanks Aniket.
>
> I'll revert the aforementioned commits in 0.12 tonight. I will leave them
> in trunk until we decide what to do.
>
>
> On Mon, Sep 30, 2013 at 3:37 PM, Aniket Mokashi <an...@gmail.com>wrote:
>
>> +1 on reverting PIG-3419 and applying it to tez branch if its blocking
>> pig-0.12 release.
>>
>> On Mon, Sep 30, 2013 at 2:42 PM, Cheolsoo Park <pi...@gmail.com>
>> wrote:
>>
>> > I am waiting for +1 from Twitter.
>> >
>> > Like Alan suggested, let's revert PIG-3419 et al in 0.12 first. Then, we
>> > can decide what to do in trunk. I volunteer to do grunt work since I am
>> the
>> > one who committed them.
>> >
>> >
>> > On Mon, Sep 30, 2013 at 2:26 PM, Rohini Palaniswamy <
>> > rohini.aditya@gmail.com
>> > > wrote:
>> >
>> > > +1. I was already asking for keeping the new API changes only in Tez
>> > branch
>> > > till it evolves and is finalized, so I have no objections to reverting
>> > it.
>> > >
>> > > Regards,
>> > > Rohini
>> > >
>> > >
>> > > On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates <ga...@hortonworks.com>
>> > wrote:
>> > >
>> > > > We should separate out two separate concerns. If I understand
>> > correctly
>> > > > we don't need any of these changes in 0.12. So we should revert
>> these
>> > > > patches from the 12 branch so that we can get it released quickly
>> in a
>> > > > backwards compatible way.
>> > > >
>> > > > We will then have plenty of time to discuss the separate question of
>> > how
>> > > > we proceed going forward (deprecated APIs or new APIs).
>> > > >
>> > > > Alan.
>> > > >
>> > > > On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote:
>> > > >
>> > > > > Hi Jeremy,
>> > > > >
>> > > > > What you're saying makes sense, and patch is welcome. ;-) But
>> > > complexity
>> > > > > comes from that there are many classes that are associated with
>> one
>> > > > > another, and it seems necessary to bring back all of them
>> together in
>> > > > order
>> > > > > to provide full backward compatibility.
>> > > > >
>> > > > > After spending many hours on the weekend, I concluded that adding
>> > more
>> > > > > workarounds (classes, methods, packages, etc) to the current code
>> > makes
>> > > > it
>> > > > > only less maintainable and readable. So I prefer a simpler
>> approach.
>> > > > >
>> > > > > For eg, we can just publish two jars - pig.jar w/ old API and
>> > > pig-new.jar
>> > > > > w/ new API - maybe not in 0.12 but in 0.13. Since we already have
>> a
>> > > > > tez-branch, we can use it to manage the new version of classes.
>> Then,
>> > > > users
>> > > > > can switch to pig-new.jar gradually in 0.13 and 0.14. When we
>> finally
>> > > > merge
>> > > > > tez-branch into trunk, we can publish a single jar again.
>> > > > >
>> > > > > Of course, this is not trivial either because we have to maintain
>> two
>> > > > > branches. But I feel that managing two branches independently is
>> > easier
>> > > > > than maintaining all sorts of workarounds for backward
>> compatibility
>> > in
>> > > > the
>> > > > > source code. In addition, we will have more flexibility in terms
>> of
>> > > > > designing new API because we will be completely free from backward
>> > > > > compatibility. No?
>> > > > >
>> > > > > Thanks,
>> > > > > Cheolsoo
>> > > > >
>> > > > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <
>> jkarn@mortardata.com>
>> > > > wrote:
>> > > > >
>> > > > >> What about the option of leaving all of the MR specific logic in
>> the
>> > > > >> original classes but marking those methods as deprecated and
>> telling
>> > > > people
>> > > > >> to switch to using a MR specific object that extends the original
>> > > class.
>> > > > >> So for example:
>> > > > >>
>> > > > >> JobStats - Reverted to being as it was before PIG-3419 but with
>> all
>> > MR
>> > > > >> specific logic deprecated.
>> > > > >> MRJobStats - Would just extend JobStats.
>> > > > >>
>> > > > >> If we did this, external software could switch their code from
>> using
>> > > > >> JobStats to MRJobStats at their own pace and without breaking
>> > against
>> > > > any
>> > > > >> specific version of Pig. After a few versions the MR specific
>> logic
>> > > > could
>> > > > >> be removed from JobStats and pushed into MRJobStats and it
>> shouldn't
>> > > > break
>> > > > >> anything for people that had made that change.
>> > > > >>
>> > > > >> I'm not familiar with all of the changes in PIG-3419 so this
>> might
>> > not
>> > > > work
>> > > > >> everywhere.
>> > > > >>
>> > > > >>
>> > > > >> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <
>> > piaozhexiu@gmail.com>
>> > > > >> wrote:
>> > > > >>
>> > > > >>> To be specific, we will need to revert all the following
>> commits in
>> > > > >> order:
>> > > > >>>
>> > > > >>>
>> > > > >>> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
>> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
>> > > > >>> Date: Fri Sep 20 22:47:29 2013 +0000
>> > > > >>>
>> > > > >>> PIG-3471: Add a base abstract class for ExecutionEngine
>> > (cheolsoo)
>> > > > >>>
>> > > > >>> git-svn-id:
>> > > > >>>
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
>> > > > >>>
>> > > > >>> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
>> > > > >>> Author: Jianyong Dai <da...@apache.org>
>> > > > >>> Date: Wed Sep 18 19:09:49 2013 +0000
>> > > > >>>
>> > > > >>> PIG-3457: Provide backward compatibility for PigStatsUtil and
>> > > > >> JobStats
>> > > > >>>
>> > > > >>> git-svn-id:
>> > > > >>>
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68
>> > > > >>>
>> > > > >>> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7
>> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
>> > > > >>> Date: Wed Sep 18 15:37:58 2013 +0000
>> > > > >>>
>> > > > >>> PIG-3464: Mark ExecType and ExecutionEngine interfaces as
>> > evolving
>> > > > >>> (cheolsoo)
>> > > > >>>
>> > > > >>> commit fd8b7cdf9292b305f02386d560c25298ab492a0b
>> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
>> > > > >>> Date: Fri Aug 30 20:04:29 2013 +0000
>> > > > >>>
>> > > > >>> PIG-3419: Pluggable Execution Engine (achalsoni81 via
>> cheolsoo)
>> > > > >>>
>> > > > >>> git-svn-id:
>> > > > >>>
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>> https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <
>> > daijy@hortonworks.com>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> Thanks Cheolsoo! My opinion is provide backward compatibility
>> for
>> > > > >>> PigStats
>> > > > >>>> is a must, otherwise it could be a havoc. I imagine PigStats is
>> > > widely
>> > > > >>> used
>> > > > >>>> by Pig users via PigRunner and PPNL interface. People use
>> PigStats
>> > > to
>> > > > >>>> collect MR job details of the Pig job. Though PigStats is
>> marked
>> > for
>> > > > >>>> Evolving, this is mostly for extending PigStats, not limiting
>> > > PigStats
>> > > > >> as
>> > > > >>>> PIG-3419 did. Even if we really need to change, we need to very
>> > well
>> > > > >>>> communicate with users over time, Pig 0.12 is not an option.
>> > > > >>>>
>> > > > >>>> PIG-3457 is trying to provide a backward compatibility way for
>> > > > >> PigStats,
>> > > > >>>> but just like Cheolsoo said, it is far from ideal. I now tend
>> to
>> > > agree
>> > > > >>>> Rohini's suggestion on PIG-3419, rollback PIG-3419, until we
>> find
>> > a
>> > > > >>> better
>> > > > >>>> way. Seems PIG-3419 is a little premature. Besides the above
>> > > mentioned
>> > > > >>>> PigStats issue, I've already found 2 additional issues:
>> > > > >>>> 1. "explain" shows unoptimized logical plan instead of
>> optimized
>> > one
>> > > > >>>> 2. HangingJobKiller is removed
>> > > > >>>>
>> > > > >>>> How does others think?
>> > > > >>>>
>> > > > >>>> Thanks,
>> > > > >>>> Daniel
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park <
>> > > piaozhexiu@gmail.com>
>> > > > >>>> wrote:
>> > > > >>>>
>> > > > >>>>> Hi devs,
>> > > > >>>>>
>> > > > >>>>> PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419>
>> broke
>> > > > >>> backward
>> > > > >>>>> compatibility for downstream applications such as Oozie, and
>> > > > >>>>> PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is
>> > > > >>>>> trying to fix it.
>> > > > >>>>>
>> > > > >>>>> In summary, we need to keep the old MR-specific JobStats and
>> > > PigStats
>> > > > >>> for
>> > > > >>>>> backward compatibility sadly because they're currently
>> exposed in
>> > > > >>> several
>> > > > >>>>> user-facing API including PigRunner, PigServer, etc. However,
>> > this
>> > > > >>>> defeats
>> > > > >>>>> the purpose of PIG-3419
>> > > > >>>>> <https://issues.apache.org/jira/browse/PIG-3419> because
>> > > > >>>>> we cannot implement non-MR execution engines in the back-end
>> if
>> > the
>> > > > >>>>> front-end API is tied to MR.
>> > > > >>>>>
>> > > > >>>>> Daniel and I were having a discussion in the jira, but it
>> seems
>> > > more
>> > > > >>>>> complicated than I thought. I am wondering whether anyone has
>> a
>> > > good
>> > > > >>>>> suggestion on how to solve it. This is blocking the 0.12
>> release.
>> > > > >>>>>
>> > > > >>>>> Thanks,
>> > > > >>>>> Cheolsoo
>> > > > >>>>>
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>> CONFIDENTIALITY NOTICE
>> > > > >>>> NOTICE: This message is intended for the use of the individual
>> or
>> > > > >> entity
>> > > > >>> to
>> > > > >>>> which it is addressed and may contain information that is
>> > > > confidential,
>> > > > >>>> privileged and exempt from disclosure under applicable law. If
>> the
>> > > > >> reader
>> > > > >>>> of this message is not the intended recipient, you are hereby
>> > > notified
>> > > > >>> that
>> > > > >>>> any printing, copying, dissemination, distribution, disclosure
>> or
>> > > > >>>> forwarding of this communication is strictly prohibited. If you
>> > have
>> > > > >>>> received this communication in error, please contact the sender
>> > > > >>> immediately
>> > > > >>>> and delete it from your system. Thank You.
>> > > > >>>>
>> > > > >>>
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >>
>> > > > >> Jeremy Karn / Lead Developer
>> > > > >> MORTAR DATA / 519 277 4391 / www.mortardata.com
>> > > > >>
>> > > >
>> > > >
>> > > > --
>> > > > CONFIDENTIALITY NOTICE
>> > > > NOTICE: This message is intended for the use of the individual or
>> > entity
>> > > to
>> > > > which it is addressed and may contain information that is
>> confidential,
>> > > > privileged and exempt from disclosure under applicable law. If the
>> > reader
>> > > > of this message is not the intended recipient, you are hereby
>> notified
>> > > that
>> > > > any printing, copying, dissemination, distribution, disclosure or
>> > > > forwarding of this communication is strictly prohibited. If you have
>> > > > received this communication in error, please contact the sender
>> > > immediately
>> > > > and delete it from your system. Thank You.
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> "...:::Aniket:::... Quetzalco@tl"
>>
>
>
Re: [Discussion] Any thoughts on PIG-3457?
Posted by Cheolsoo Park <pi...@gmail.com>.
Thanks Aniket.
I'll revert the aforementioned commits in 0.12 tonight. I will leave them
in trunk until we decide what to do.
On Mon, Sep 30, 2013 at 3:37 PM, Aniket Mokashi <an...@gmail.com> wrote:
> +1 on reverting PIG-3419 and applying it to tez branch if its blocking
> pig-0.12 release.
>
> On Mon, Sep 30, 2013 at 2:42 PM, Cheolsoo Park <pi...@gmail.com>
> wrote:
>
> > I am waiting for +1 from Twitter.
> >
> > Like Alan suggested, let's revert PIG-3419 et al in 0.12 first. Then, we
> > can decide what to do in trunk. I volunteer to do grunt work since I am
> the
> > one who committed them.
> >
> >
> > On Mon, Sep 30, 2013 at 2:26 PM, Rohini Palaniswamy <
> > rohini.aditya@gmail.com
> > > wrote:
> >
> > > +1. I was already asking for keeping the new API changes only in Tez
> > branch
> > > till it evolves and is finalized, so I have no objections to reverting
> > it.
> > >
> > > Regards,
> > > Rohini
> > >
> > >
> > > On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates <ga...@hortonworks.com>
> > wrote:
> > >
> > > > We should separate out two separate concerns. If I understand
> > correctly
> > > > we don't need any of these changes in 0.12. So we should revert
> these
> > > > patches from the 12 branch so that we can get it released quickly in
> a
> > > > backwards compatible way.
> > > >
> > > > We will then have plenty of time to discuss the separate question of
> > how
> > > > we proceed going forward (deprecated APIs or new APIs).
> > > >
> > > > Alan.
> > > >
> > > > On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote:
> > > >
> > > > > Hi Jeremy,
> > > > >
> > > > > What you're saying makes sense, and patch is welcome. ;-) But
> > > complexity
> > > > > comes from that there are many classes that are associated with one
> > > > > another, and it seems necessary to bring back all of them together
> in
> > > > order
> > > > > to provide full backward compatibility.
> > > > >
> > > > > After spending many hours on the weekend, I concluded that adding
> > more
> > > > > workarounds (classes, methods, packages, etc) to the current code
> > makes
> > > > it
> > > > > only less maintainable and readable. So I prefer a simpler
> approach.
> > > > >
> > > > > For eg, we can just publish two jars - pig.jar w/ old API and
> > > pig-new.jar
> > > > > w/ new API - maybe not in 0.12 but in 0.13. Since we already have a
> > > > > tez-branch, we can use it to manage the new version of classes.
> Then,
> > > > users
> > > > > can switch to pig-new.jar gradually in 0.13 and 0.14. When we
> finally
> > > > merge
> > > > > tez-branch into trunk, we can publish a single jar again.
> > > > >
> > > > > Of course, this is not trivial either because we have to maintain
> two
> > > > > branches. But I feel that managing two branches independently is
> > easier
> > > > > than maintaining all sorts of workarounds for backward
> compatibility
> > in
> > > > the
> > > > > source code. In addition, we will have more flexibility in terms of
> > > > > designing new API because we will be completely free from backward
> > > > > compatibility. No?
> > > > >
> > > > > Thanks,
> > > > > Cheolsoo
> > > > >
> > > > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn <
> jkarn@mortardata.com>
> > > > wrote:
> > > > >
> > > > >> What about the option of leaving all of the MR specific logic in
> the
> > > > >> original classes but marking those methods as deprecated and
> telling
> > > > people
> > > > >> to switch to using a MR specific object that extends the original
> > > class.
> > > > >> So for example:
> > > > >>
> > > > >> JobStats - Reverted to being as it was before PIG-3419 but with
> all
> > MR
> > > > >> specific logic deprecated.
> > > > >> MRJobStats - Would just extend JobStats.
> > > > >>
> > > > >> If we did this, external software could switch their code from
> using
> > > > >> JobStats to MRJobStats at their own pace and without breaking
> > against
> > > > any
> > > > >> specific version of Pig. After a few versions the MR specific
> logic
> > > > could
> > > > >> be removed from JobStats and pushed into MRJobStats and it
> shouldn't
> > > > break
> > > > >> anything for people that had made that change.
> > > > >>
> > > > >> I'm not familiar with all of the changes in PIG-3419 so this might
> > not
> > > > work
> > > > >> everywhere.
> > > > >>
> > > > >>
> > > > >> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park <
> > piaozhexiu@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> To be specific, we will need to revert all the following commits
> in
> > > > >> order:
> > > > >>>
> > > > >>>
> > > > >>> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982
> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
> > > > >>> Date: Fri Sep 20 22:47:29 2013 +0000
> > > > >>>
> > > > >>> PIG-3471: Add a base abstract class for ExecutionEngine
> > (cheolsoo)
> > > > >>>
> > > > >>> git-svn-id:
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68
> > > > >>>
> > > > >>> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1
> > > > >>> Author: Jianyong Dai <da...@apache.org>
> > > > >>> Date: Wed Sep 18 19:09:49 2013 +0000
> > > > >>>
> > > > >>> PIG-3457: Provide backward compatibility for PigStatsUtil and
> > > > >> JobStats
> > > > >>>
> > > > >>> git-svn-id:
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68
> > > > >>>
> > > > >>> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7
> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
> > > > >>> Date: Wed Sep 18 15:37:58 2013 +0000
> > > > >>>
> > > > >>> PIG-3464: Mark ExecType and ExecutionEngine interfaces as
> > evolving
> > > > >>> (cheolsoo)
> > > > >>>
> > > > >>> commit fd8b7cdf9292b305f02386d560c25298ab492a0b
> > > > >>> Author: Cheolsoo Park <ch...@apache.org>
> > > > >>> Date: Fri Aug 30 20:04:29 2013 +0000
> > > > >>>
> > > > >>> PIG-3419: Pluggable Execution Engine (achalsoni81 via
> cheolsoo)
> > > > >>>
> > > > >>> git-svn-id:
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai <
> > daijy@hortonworks.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Thanks Cheolsoo! My opinion is provide backward compatibility
> for
> > > > >>> PigStats
> > > > >>>> is a must, otherwise it could be a havoc. I imagine PigStats is
> > > widely
> > > > >>> used
> > > > >>>> by Pig users via PigRunner and PPNL interface. People use
> PigStats
> > > to
> > > > >>>> collect MR job details of the Pig job. Though PigStats is marked
> > for
> > > > >>>> Evolving, this is mostly for extending PigStats, not limiting
> > > PigStats
> > > > >> as
> > > > >>>> PIG-3419 did. Even if we really need to change, we need to very
> > well
> > > > >>>> communicate with users over time, Pig 0.12 is not an option.
> > > > >>>>
> > > > >>>> PIG-3457 is trying to provide a backward compatibility way for
> > > > >> PigStats,
> > > > >>>> but just like Cheolsoo said, it is far from ideal. I now tend to
> > > agree
> > > > >>>> Rohini's suggestion on PIG-3419, rollback PIG-3419, until we
> find
> > a
> > > > >>> better
> > > > >>>> way. Seems PIG-3419 is a little premature. Besides the above
> > > mentioned
> > > > >>>> PigStats issue, I've already found 2 additional issues:
> > > > >>>> 1. "explain" shows unoptimized logical plan instead of optimized
> > one
> > > > >>>> 2. HangingJobKiller is removed
> > > > >>>>
> > > > >>>> How does others think?
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Daniel
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park <
> > > piaozhexiu@gmail.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Hi devs,
> > > > >>>>>
> > > > >>>>> PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419>
> broke
> > > > >>> backward
> > > > >>>>> compatibility for downstream applications such as Oozie, and
> > > > >>>>> PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is
> > > > >>>>> trying to fix it.
> > > > >>>>>
> > > > >>>>> In summary, we need to keep the old MR-specific JobStats and
> > > PigStats
> > > > >>> for
> > > > >>>>> backward compatibility sadly because they're currently exposed
> in
> > > > >>> several
> > > > >>>>> user-facing API including PigRunner, PigServer, etc. However,
> > this
> > > > >>>> defeats
> > > > >>>>> the purpose of PIG-3419
> > > > >>>>> <https://issues.apache.org/jira/browse/PIG-3419> because
> > > > >>>>> we cannot implement non-MR execution engines in the back-end if
> > the
> > > > >>>>> front-end API is tied to MR.
> > > > >>>>>
> > > > >>>>> Daniel and I were having a discussion in the jira, but it seems
> > > more
> > > > >>>>> complicated than I thought. I am wondering whether anyone has a
> > > good
> > > > >>>>> suggestion on how to solve it. This is blocking the 0.12
> release.
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>> Cheolsoo
> > > > >>>>>
> > > > >>>>
> > > > >>>> --
> > > > >>>> CONFIDENTIALITY NOTICE
> > > > >>>> NOTICE: This message is intended for the use of the individual
> or
> > > > >> entity
> > > > >>> to
> > > > >>>> which it is addressed and may contain information that is
> > > > confidential,
> > > > >>>> privileged and exempt from disclosure under applicable law. If
> the
> > > > >> reader
> > > > >>>> of this message is not the intended recipient, you are hereby
> > > notified
> > > > >>> that
> > > > >>>> any printing, copying, dissemination, distribution, disclosure
> or
> > > > >>>> forwarding of this communication is strictly prohibited. If you
> > have
> > > > >>>> received this communication in error, please contact the sender
> > > > >>> immediately
> > > > >>>> and delete it from your system. Thank You.
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Jeremy Karn / Lead Developer
> > > > >> MORTAR DATA / 519 277 4391 / www.mortardata.com
> > > > >>
> > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>
>
>
> --
> "...:::Aniket:::... Quetzalco@tl"
>