[Discussion] Keep or revert PIG-3419 in trunk?

[Cheolsoo Park <pi...@gmail.com>]

Hi devs,

This is a follow-up discussion about how to resolve the backward
incompatibility of PIG-3419 (Pluggable execution engine). Per the previous
discussion <http://search-hadoop.com/m/wYz6hz9CoE>, I reverted it in 0.12
but kept it in trunk. As we keep committing more changes into trunk, it
gets harder to back out PIG-3419 cleanly. So I suggest we should make a
decision sooner rather than later.

The crux of the problem is as follows:

PIG-3419 removes all the MR-specific things from JobStats. However,
PigRunner and PigServer returns PigStats that in turn exposes JobStats to
end users. So changing JobStats breaks backward compatibility for
downstream projects such as Oozie. While changing the semantics of JobStats
is acceptable, we must provide a deprecation path so that end users can
upgrade their applications smoothly.

Proposed solutions:

1) Provide backward compatibility in source code: Maybe possible, but no
one has come up with a clean solution. For eg, I failed. :(

2) Publish two jars: We keep PIG-3419 only in tez-branch and publish two
jars (one for old API and one for new API) for a couple of future releases.
If we do this, we're going to revert PIG-3419 in trunk.

What do you think?

Thanks,
Cheolsoo

Re: [Discussion] Keep or revert PIG-3419 in trunk?

[Cheolsoo Park <pi...@gmail.com>]

To close the loop, here is the current status of issues that PIG-3419
introduced-

1) Backward incompatibility: Daniel provided a clever solution (PIG-3520).
In summary, JobStats and PigStatsUtil will keep MR-specific methods as
abstract methods. All the sub-classes that extend them (e.g.
SimpleJobStats, TezJobStats, etc) must implement those methods. Non-MR
classes will simply throw a "not implemented" exception for those methods.

2) HangingJobKiller: PIG-3524 brings it back to the launcher. Pending
review.

3) Explain shows unoptimized plan: PIG-3508 keeps track of the issue. To be
fixed.

4) PigStats.get() and ScriptState.get() return MR-specific objects:
PIG-3525 keeps track of the issue. To be fixed.

Given the current status, I think we can keep PIG-3419 in trunk. Please let
me know if anyone thinks otherwise.

Thanks,
Cheolsoo

On Tue, Oct 15, 2013 at 1:54 PM, Daniel Dai <da...@hortonworks.com> wrote:

> I still want to try a little bit. Give me one more day.
>
> Thanks,
> Daniel
>
>
> On Tue, Oct 15, 2013 at 12:40 PM, Jeremy Karn <jk...@mortardata.com>
> wrote:
>
> > I'm +1 on reverting PIG-3419 on trunk and then trying to redo it in a
> > better way.  Either way it would be good to get this resolved soon.
> >
> >
> > On Mon, Oct 7, 2013 at 5:49 PM, Daniel Dai <da...@hortonworks.com>
> wrote:
> >
> > > Let me first revert PIG-3457, that complicates things. I will try to
> > > explore for a solution in next the few days. But if we cannot settle
> > > quickly, I would suggest to revert PIG-3419 on trunk first, and redo it
> > > later in a better way. Otherwise, it will be very hard to revert.
> > >
> > > Thanks,
> > > Daniel
> > >
> > >
> > > On Mon, Oct 7, 2013 at 12:05 PM, Cheolsoo Park <pi...@gmail.com>
> > > wrote:
> > >
> > > > Hi devs,
> > > >
> > > > This is a follow-up discussion about how to resolve the backward
> > > > incompatibility of PIG-3419 (Pluggable execution engine). Per the
> > > previous
> > > > discussion <http://search-hadoop.com/m/wYz6hz9CoE>, I reverted it in
> > > 0.12
> > > > but kept it in trunk. As we keep committing more changes into trunk,
> it
> > > > gets harder to back out PIG-3419 cleanly. So I suggest we should
> make a
> > > > decision sooner rather than later.
> > > >
> > > > The crux of the problem is as follows:
> > > >
> > > > PIG-3419 removes all the MR-specific things from JobStats. However,
> > > > PigRunner and PigServer returns PigStats that in turn exposes
> JobStats
> > to
> > > > end users. So changing JobStats breaks backward compatibility for
> > > > downstream projects such as Oozie. While changing the semantics of
> > > JobStats
> > > > is acceptable, we must provide a deprecation path so that end users
> can
> > > > upgrade their applications smoothly.
> > > >
> > > > Proposed solutions:
> > > >
> > > > 1) Provide backward compatibility in source code: Maybe possible, but
> > no
> > > > one has come up with a clean solution. For eg, I failed. :(
> > > >
> > > > 2) Publish two jars: We keep PIG-3419 only in tez-branch and publish
> > two
> > > > jars (one for old API and one for new API) for a couple of future
> > > releases.
> > > > If we do this, we're going to revert PIG-3419 in trunk.
> > > >
> > > > What do you think?
> > > >
> > > > Thanks,
> > > > Cheolsoo
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
> >
> >
> > --
> >
> > Jeremy Karn / Lead Developer
> > MORTAR DATA / 519 277 4391 / www.mortardata.com
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: [Discussion] Keep or revert PIG-3419 in trunk?

[Daniel Dai <da...@hortonworks.com>]

I still want to try a little bit. Give me one more day.

Thanks,
Daniel


On Tue, Oct 15, 2013 at 12:40 PM, Jeremy Karn <jk...@mortardata.com> wrote:

> I'm +1 on reverting PIG-3419 on trunk and then trying to redo it in a
> better way.  Either way it would be good to get this resolved soon.
>
>
> On Mon, Oct 7, 2013 at 5:49 PM, Daniel Dai <da...@hortonworks.com> wrote:
>
> > Let me first revert PIG-3457, that complicates things. I will try to
> > explore for a solution in next the few days. But if we cannot settle
> > quickly, I would suggest to revert PIG-3419 on trunk first, and redo it
> > later in a better way. Otherwise, it will be very hard to revert.
> >
> > Thanks,
> > Daniel
> >
> >
> > On Mon, Oct 7, 2013 at 12:05 PM, Cheolsoo Park <pi...@gmail.com>
> > wrote:
> >
> > > Hi devs,
> > >
> > > This is a follow-up discussion about how to resolve the backward
> > > incompatibility of PIG-3419 (Pluggable execution engine). Per the
> > previous
> > > discussion <http://search-hadoop.com/m/wYz6hz9CoE>, I reverted it in
> > 0.12
> > > but kept it in trunk. As we keep committing more changes into trunk, it
> > > gets harder to back out PIG-3419 cleanly. So I suggest we should make a
> > > decision sooner rather than later.
> > >
> > > The crux of the problem is as follows:
> > >
> > > PIG-3419 removes all the MR-specific things from JobStats. However,
> > > PigRunner and PigServer returns PigStats that in turn exposes JobStats
> to
> > > end users. So changing JobStats breaks backward compatibility for
> > > downstream projects such as Oozie. While changing the semantics of
> > JobStats
> > > is acceptable, we must provide a deprecation path so that end users can
> > > upgrade their applications smoothly.
> > >
> > > Proposed solutions:
> > >
> > > 1) Provide backward compatibility in source code: Maybe possible, but
> no
> > > one has come up with a clean solution. For eg, I failed. :(
> > >
> > > 2) Publish two jars: We keep PIG-3419 only in tez-branch and publish
> two
> > > jars (one for old API and one for new API) for a couple of future
> > releases.
> > > If we do this, we're going to revert PIG-3419 in trunk.
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > Cheolsoo
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
>
> Jeremy Karn / Lead Developer
> MORTAR DATA / 519 277 4391 / www.mortardata.com
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [Discussion] Keep or revert PIG-3419 in trunk?

[Jeremy Karn <jk...@mortardata.com>]

I'm +1 on reverting PIG-3419 on trunk and then trying to redo it in a
better way.  Either way it would be good to get this resolved soon.


On Mon, Oct 7, 2013 at 5:49 PM, Daniel Dai <da...@hortonworks.com> wrote:

> Let me first revert PIG-3457, that complicates things. I will try to
> explore for a solution in next the few days. But if we cannot settle
> quickly, I would suggest to revert PIG-3419 on trunk first, and redo it
> later in a better way. Otherwise, it will be very hard to revert.
>
> Thanks,
> Daniel
>
>
> On Mon, Oct 7, 2013 at 12:05 PM, Cheolsoo Park <pi...@gmail.com>
> wrote:
>
> > Hi devs,
> >
> > This is a follow-up discussion about how to resolve the backward
> > incompatibility of PIG-3419 (Pluggable execution engine). Per the
> previous
> > discussion <http://search-hadoop.com/m/wYz6hz9CoE>, I reverted it in
> 0.12
> > but kept it in trunk. As we keep committing more changes into trunk, it
> > gets harder to back out PIG-3419 cleanly. So I suggest we should make a
> > decision sooner rather than later.
> >
> > The crux of the problem is as follows:
> >
> > PIG-3419 removes all the MR-specific things from JobStats. However,
> > PigRunner and PigServer returns PigStats that in turn exposes JobStats to
> > end users. So changing JobStats breaks backward compatibility for
> > downstream projects such as Oozie. While changing the semantics of
> JobStats
> > is acceptable, we must provide a deprecation path so that end users can
> > upgrade their applications smoothly.
> >
> > Proposed solutions:
> >
> > 1) Provide backward compatibility in source code: Maybe possible, but no
> > one has come up with a clean solution. For eg, I failed. :(
> >
> > 2) Publish two jars: We keep PIG-3419 only in tez-branch and publish two
> > jars (one for old API and one for new API) for a couple of future
> releases.
> > If we do this, we're going to revert PIG-3419 in trunk.
> >
> > What do you think?
> >
> > Thanks,
> > Cheolsoo
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>



-- 

Jeremy Karn / Lead Developer
MORTAR DATA / 519 277 4391 / www.mortardata.com

Re: [Discussion] Keep or revert PIG-3419 in trunk?

[Daniel Dai <da...@hortonworks.com>]

Let me first revert PIG-3457, that complicates things. I will try to
explore for a solution in next the few days. But if we cannot settle
quickly, I would suggest to revert PIG-3419 on trunk first, and redo it
later in a better way. Otherwise, it will be very hard to revert.

Thanks,
Daniel


On Mon, Oct 7, 2013 at 12:05 PM, Cheolsoo Park <pi...@gmail.com> wrote:

> Hi devs,
>
> This is a follow-up discussion about how to resolve the backward
> incompatibility of PIG-3419 (Pluggable execution engine). Per the previous
> discussion <http://search-hadoop.com/m/wYz6hz9CoE>, I reverted it in 0.12
> but kept it in trunk. As we keep committing more changes into trunk, it
> gets harder to back out PIG-3419 cleanly. So I suggest we should make a
> decision sooner rather than later.
>
> The crux of the problem is as follows:
>
> PIG-3419 removes all the MR-specific things from JobStats. However,
> PigRunner and PigServer returns PigStats that in turn exposes JobStats to
> end users. So changing JobStats breaks backward compatibility for
> downstream projects such as Oozie. While changing the semantics of JobStats
> is acceptable, we must provide a deprecation path so that end users can
> upgrade their applications smoothly.
>
> Proposed solutions:
>
> 1) Provide backward compatibility in source code: Maybe possible, but no
> one has come up with a clean solution. For eg, I failed. :(
>
> 2) Publish two jars: We keep PIG-3419 only in tez-branch and publish two
> jars (one for old API and one for new API) for a couple of future releases.
> If we do this, we're going to revert PIG-3419 in trunk.
>
> What do you think?
>
> Thanks,
> Cheolsoo
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.