Re: OOZIE-1311

[Robert Kanter <rk...@cloudera.com>]

Virag,
I talked more with Alejandro about the sharelib and the launcher jar.

It would be good if we could eventually get rid of the launcher jar
completely because:
- less files would be copied to HDFS so it should be faster
- it would fix an issue where some linux distress cleanup the temp dir and
delete the launcher jar

The contents of the launcher jar could be moved to the Oozie sharelib
(/share/lib/oozie) which IIRC currently only has a json jar file.

As for backwards compatibility, we could do something like what you
suggested for the other sharelibs where the webapp module includes them.
 Then we'd add some property to oozie-site that would switch between the
old behavior with the launcher jar (sharelib is not required) and the new
behavior without the launcher jar (sharelib is required).  A new major
release, like Oozie 4, would be a good time to introduce this change as the
default; users can use the property to go back to the old behavior.  Also,
if set to use the new behavior, we should make oozie.use.system.libpath in
the job.properties default to true because the sharelib would be required,
and if set to the old behavior, it would default to false.

Thoughts?

thanks
- Robert





On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <vi...@yahoo-inc.com> wrote:

> Thanks Robert. Created OOZIE-1341
>
> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>
> >I think I follow what you're saying; assuming everything works correctly,
> >I
> >think that sounds good.  Can you make a JIRA with that and put up a patch;
> >that might be easier to see how this would work.
> >
> >thanks
> >- Robert
> >
> >
> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com>
> >wrote:
> >
> >> We can keep the Main classes in the share lib, but have the action
> >> executors package the main classes for shipping it to
> >> Launcher (basically again having the method getLauncherClasses() in
> >>action
> >> executors). Also the oozie-sharelib.jar containing main classes
> >> should be part of web-app.
> >> This will improve the building/testing env as classes are in isolated
> >> sharelibs but will make the share lib optional as main classes are part
> >>of
> >> oozie-server.
> >> Can we do this change to trunk as this is blocking our QE's to pick
> >>builds
> >> due to share lib incompatibility?
> >>
> >> We can still do what oozie-1318 is trying to achieve. For, e.g with Hcat
> >> integration,
> >> there can be multiple overridable implementation of URIHandlers that can
> >> be shipped to launcher.
> >> We can do the same for other main classes if required.
> >>
> >>
> >> Thanks,
> >> Virag
> >>
> >>
> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
> >>
> >> >That's true, its primarily a building/testing improvement.  And from a
> >> >design perspective, it could be considered "cleaner" to not have the
> >> >launcher jar.
> >> >
> >> >Your idea sounds like a good compromise between backwards compatibility
> >> >and
> >> >improving the building/testing env.  How would we do that?
> >> >
> >> >As a side note, there's also a new feature with the Main classes being
> >>in
> >> >the sharelib; users can now override the Main class that Oozie will use
> >> >for
> >> >the action because its no longer in the launcher jar.  I think this
> >>could
> >> >still be a useful feature, but I'm guessing that if we do the
> >>compromise,
> >> >this would go away?  In any case, OOZIE-1318 is to make the Main
> >> >overridable with a config, which is probably cleaner and more flexible
> >> >than
> >> >replacing a jar file.
> >> >
> >> >thanks
> >> >- Robert
> >> >
> >> >
> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <vi...@yahoo-inc.com>
> >> >wrote:
> >> >
> >> >> That is a good suggestion for preserving compatibility, Robert.
> >> >>
> >> >> Another question is what is the main purpose of OOZIE-1311? From JIRA
> >> >> description, the primary
> >> >> advantage of moving Main classes from core to share lib is it can
> >>help
> >> >>in
> >> >> preventing dependencies conflicts
> >> >> (different versions of antlr's for pig, hive). However, that problem
> >> >>only
> >> >> exists while building/testing Oozie.
> >> >> If that is the case, main classes can be moved to share lib but they
> >>can
> >> >> be bundled in oozie-server itself.
> >> >> In that case, the deployment doesn't change and share lib can still
> >>be
> >> >> optional
> >> >> as oozie server would ship those classes (same as what was happening
> >> >> before). And,
> >> >> advantage of 1311 for building/testing will be retained
> >> >>
> >> >> The main concern is having share lib as a required installation will
> >> >> complicate deployment.
> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't
> >>provide
> >> >> inbuilt support for that.
> >> >> Even though end-users wouldn't notice the change, it will be a big
> >> >>change
> >> >> for system admins installing oozie
> >> >> So even though share lib is nice feature to have, keeping it optional
> >> >> seems better to me as there are other ways
> >> >> Of including jars and which were introduced much before share lib.
> >> >>
> >> >> Thanks,
> >> >> Virag
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
> >> >>
> >> >> >Alejandro had another suggestion to prevent breaking stuff for
> >>existing
> >> >> >apps where the sharelib isn't used.  Assuming the sharelib is
> >> >>installed,
> >> >> >if
> >> >> >the user doesn't set oozie.use.system.libpath to true, then Oozie
> >>would
> >> >> >only include the oozie-*.jar from the sharelib (the one with the
> >>Main)
> >> >>in
> >> >> >it; if its set to true, then Oozie would include all of the jars in
> >>the
> >> >> >sharelib.  This would allow users to still use their own lib dir
> >>while
> >> >> >still getting the Main into the classpath.
> >> >> >
> >> >> >- Robert
> >> >> >
> >> >> >
> >> >> >
> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
> >> >> ><tu...@gmail.com>wrote:
> >> >> >
> >> >> >> virag, if your oozie servers dont have sharelibs currently
> >>installed
> >> >> >>then
> >> >> >> unstalling sharelibs as part of an update would not break ant
> >>running
> >> >> >>app.
> >> >> >> regarding the requirement of enabling the sharelibs to have the
> >> >>mains,
> >> >> >>you
> >> >> >> have a point. we need to figure out how to get those in the
> >> >>classpath if
> >> >> >> the user is not using the sharelibs. i'd suggest we leave this
> >> >> >> patch in trunk for now and we move it to a release branch once we
> >> >>sorted
> >> >> >> out a way of solving this incompatibility.
> >> >> >>
> >> >> >> thanks
> >> >> >>
> >> >> >> Alejandro
> >> >> >> (phone typing)
> >> >> >>
> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <vi...@yahoo-inc.com>
> >> >>wrote:
> >> >> >>
> >> >> >> >
> >> >> >> > Tucu, Thanks for the reply and explanation.
> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible
> >>option
> >> >>as
> >> >> >>we
> >> >> >> > cannot expect all the running actions to complete before Oozie
> >> >> >>restart.
> >> >> >> > Hot upgrade is a good option, but we will need time to
> >>implement it
> >> >> >>as we
> >> >> >> > currently don't have sharelib as part of our deployment.
> >> >> >> > So can we deprecate the refactoring of launcher classes in this
> >> >> >>release?
> >> >> >> > Also, all the launcher classes are only added to distributed
> >>cache
> >> >>if
> >> >> >>the
> >> >> >> > workflow is configured to use system lib path
> >> >> >> > (oozie.use.system.libpath=true)
> >> >> >> > We shouldn't mandate the workflow to have this property as they
> >>are
> >> >> >> > oozie's launcher classes.
> >> >> >> >
> >> >> >> > Thanks,
> >> >> >> > Virag
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tu...@gmail.com>
> >>wrote:
> >> >> >> >
> >> >> >> >> Virag,
> >> >> >> >>
> >> >> >> >> On sharelib being required, yes you are correct. For this we
> >> >>should:
> >> >> >> >>
> >> >> >> >> * Make 100% clear in the quick-start/install docs that the
> >> >>sharelib
> >> >> >>is
> >> >> >> >> REQUIRED.
> >> >> >> >>
> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir
> >>exists,
> >> >> >>else
> >> >> >> >> fail
> >> >> >> >> to start.
> >> >> >> >>
> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs.
> >> >>Depending on
> >> >> >> the
> >> >> >> >> type of upgrade this may be an issue.
> >> >> >> >>
> >> >> >> >> * If you are upgrading an oozie server fix that does not change
> >> >>the
> >> >> >> >> sharelib files, this is not an issue and you can just shutdown
> >>the
> >> >> >>oozie
> >> >> >> >> server with in-flight jobs.
> >> >> >> >>
> >> >> >> >> * If you are upgrading an oozie server fix that involves
> >>sharelib
> >> >> >>files
> >> >> >> >> changes, then you have 2 options:
> >> >> >> >>
> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait
> >> >>till
> >> >> >>all
> >> >> >> >> running actions end, then shutdown oozie server, upgrade oozie
> >> >>server
> >> >> >> and
> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In this
> >> >>case
> >> >> >>all
> >> >> >> >> new WF actions will use the new sharelib.
> >> >> >> >>
> >> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml
> >> >>sharelib
> >> >> >> >> location to point to a new directory. upgrade the oozie server.
> >> >> >>install
> >> >> >> >> the
> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does not
> >> >> >>exist).
> >> >> >> >> start the oozie server. In this case all running WF actions
> >>will
> >> >> >> continue
> >> >> >> >> running with no issues as the JARs in the distributed cache
> >>have
> >> >>not
> >> >> >> been
> >> >> >> >> touch. All new WF actions will start using the new sharelib.
> >> >> >> >>
> >> >> >> >> Note that this sharelib upgrade protocol is not introduced by
> >> >> >>requiring
> >> >> >> >> sharelib, it is required if you have applications that use
> >> >>sharelib
> >> >> >> today.
> >> >> >> >>
> >> >> >> >> Does this address your concerns?
> >> >> >> >>
> >> >> >> >> Thanks.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
> >> >><vi...@yahoo-inc.com>
> >> >> >> >> wrote:
> >> >> >> >>
> >> >> >> >>> Hi,
> >> >> >> >>>
> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to move all
> >>the
> >> >> >> >>> launcher
> >> >> >> >>> classes like PigMain, HiveMain etc. to  their respective
> >> >>sharelibs.
> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before
> >>shared
> >> >>lib
> >> >> >> was
> >> >> >> >>> optional as users could bundle jars with their workflow
> >> >>application.
> >> >> >> >>> So always requiring shared lib seems to introduce 2 problems:
> >> >> >> >>>
> >> >> >> >>>  1.  The current deployments which don't use action shared lib
> >> >>will
> >> >> >> >>> fail.
> >> >> >> >>> So, probably we should deprecate the current behavior.
> >> >> >> >>>
> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job if
> >>the
> >> >> >>files
> >> >> >> >>> in
> >> >> >> >>> DC are updated on hdfs while the hadoop job is running. So,
> >>when
> >> >> >>Oozie
> >> >> >> >>> is
> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of
> >> >> >> >>>             deployment, hadoop  will fail the existing jobs
> >>for
> >> >> >>which
> >> >> >> >>> the
> >> >> >> >>> timestamp of  the file on hdfs doesn't match the timestamp of
> >>its
> >> >> >>copy
> >> >> >> >>> in
> >> >> >> >>> the job's DC.
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> Thanks,
> >> >> >> >>> Virag
> >> >> >> >
> >> >> >>
> >> >>
> >> >>
> >>
> >>
>
>

Re: OOZIE-1311

[Robert Kanter <rk...@cloudera.com>]

Oh, we can use trunk as Oozie 4.0.  In that case I would leave the default
as true (to use the launcher jar) in trunk and create a JIRA to change it
to false for 5.0.  Easier for me if we just use trunk :)

thanks
- Robert


On Thu, Jun 6, 2013 at 10:54 AM, Rohini Palaniswamy <rohini.aditya@gmail.com
> wrote:

> Robert,
>    Do you want to spend effort to rebase and put this patch in branch-4
> also? I was wondering since we were thinking of releasing trunk with SLA
> work along with HCat as 4.0 instead of releasing the current branch-4 to
> consolidate the database changes. If you would like to release branch-4,
> then we need to do the OozieDBCLI table changes for SLA work as 5.0 and
> change trunk to 5.0 then.
>
>
> Regards,
> Rohini
>
>
> On Thu, Jun 6, 2013 at 10:04 AM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > It took forever, but I finally got the refactoring done; I just uploaded
> a
> > patch at OOZIE-1315 <https://issues.apache.org/jira/browse/OOZIE-1315>.
> >  Please take a look.
> >
> > The patch has the default set to true (use the launcher jar).  When I
> > commit it I can have the version in branch-4 use true and the version in
> > trunk use false.
> >
> > thanks
> > - Robert
> >
> >
> > On Mon, May 6, 2013 at 3:06 PM, Rohini Palaniswamy
> > <ro...@gmail.com>wrote:
> >
> > > Tucu,
> > >    That sounds good. Can we have the changes go in trunk first and once
> > we
> > > validate that everything is fine, can we then port all the patches to
> > > branch-4.0?
> > >
> > > Thanks,
> > > Rohini
> > >
> > >
> > > On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <tucu@cloudera.com
> > > >wrote:
> > >
> > > > Rohini,
> > > >
> > > > I understand you don't want to destabilize Oozie 4. Still, as we have
> > no
> > > > done a 4 release yet, I think it is a good time to introduce this
> > > feature.
> > > >
> > > > How about the following (similar to what we did for command
> > > interruptions)?
> > > >
> > > > We do as Robert proposed in his last email, but keeping the default
> > value
> > > > on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON.
> > > >
> > > > This would mean you don't have to change anything in your deployment
> > > > protocol for Oozie 4.
> > > >
> > > > Thx
> > > >
> > > >
> > > >
> > > > On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy <
> > > > rohini.aditya@gmail.com> wrote:
> > > >
> > > >> Robert,
> > > >> > A new major release, like Oozie 4, would be a good time to
> introduce
> > > >> this change as the default;
> > > >>
> > > >>    We already have Oozie 4 deployed in some clusters and it is
> stable
> > > >> now. We are waiting for users to validate Oozie-HCat integration
> > before
> > > we
> > > >> deploy them in production clusters and call out a release in Apache.
> > So
> > > it
> > > >> would be good to introduce this in the next release which will also
> > be a
> > > >> big release anyway.
> > > >>
> > > >> Regards,
> > > >> Rohini
> > > >>
> > > >>
> > > >> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkanter@cloudera.com
> > > >wrote:
> > > >>
> > > >>> Sorry, I realize that I wasn't clear on that point.  The Main
> classes
> > > >>> would
> > > >>> go in their respective sharelib (e.g. PigMain goes in
> share/lib/pig/)
> > > but
> > > >>> the other few classes that all actions use in the launcher jar
> (e.g.
> > > >>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie),
> > which
> > > >>> IIRC is already included when you use sharelibs (for example, when
> > you
> > > >>> use
> > > >>> the sharelib with the Pig action, it includes both the
> share/lib/pig
> > > and
> > > >>> share/lib/oozie).
> > > >>>
> > > >>> thanks
> > > >>> - Robert
> > > >>>
> > > >>>
> > > >>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <virag@yahoo-inc.com
> >
> > > >>> wrote:
> > > >>>
> > > >>> >  Hi Robert,
> > > >>> >
> > > >>> >  I didn't understand completely. Are you suggesting to have all
> > > action
> > > >>> > specific main classes in /share/lib/oozie instead of their
> > respective
> > > >>> > action sharelibs?
> > > >>> >
> > > >>> >  Thanks,
> > > >>> > Virag
> > > >>> >
> > > >>> >   From: Robert Kanter <rk...@cloudera.com>
> > > >>> > Date: Monday, May 6, 2013 12:47 PM
> > > >>> > To: "dev@oozie.apache.org" <de...@oozie.apache.org>, Virag
> Kothari <
> > > >>> > virag@yahoo-inc.com>, Alejandro Abdelnur <tu...@cloudera.com>
> > > >>> > Subject: Re: OOZIE-1311
> > > >>> >
> > > >>> >   Virag,
> > > >>> > I talked more with Alejandro about the sharelib and the launcher
> > jar.
> > > >>> >
> > > >>> > It would be good if we could eventually get rid of the launcher
> jar
> > > >>> > completely because:
> > > >>> >  - less files would be copied to HDFS so it should be faster
> > > >>> > - it would fix an issue where some linux distress cleanup the
> temp
> > > dir
> > > >>> and
> > > >>> > delete the launcher jar
> > > >>> >
> > > >>> >  The contents of the launcher jar could be moved to the Oozie
> > > sharelib
> > > >>> > (/share/lib/oozie) which IIRC currently only has a json jar file.
> > > >>> >
> > > >>> >  As for backwards compatibility, we could do something like what
> > you
> > > >>> > suggested for the other sharelibs where the webapp module
> includes
> > > >>> them.
> > > >>> >  Then we'd add some property to oozie-site that would switch
> > between
> > > >>> the
> > > >>> > old behavior with the launcher jar (sharelib is not required) and
> > the
> > > >>> new
> > > >>> > behavior without the launcher jar (sharelib is required).  A new
> > > major
> > > >>> > release, like Oozie 4, would be a good time to introduce this
> > change
> > > >>> as the
> > > >>> > default; users can use the property to go back to the old
> behavior.
> > > >>>  Also,
> > > >>> > if set to use the new behavior, we should make
> > > >>> oozie.use.system.libpath in
> > > >>> > the job.properties default to true because the sharelib would be
> > > >>> required,
> > > >>> > and if set to the old behavior, it would default to false.
> > > >>> >
> > > >>> >  Thoughts?
> > > >>> >
> > > >>> >  thanks
> > > >>> > - Robert
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <
> > virag@yahoo-inc.com
> > > >>> >wrote:
> > > >>> >
> > > >>> >> Thanks Robert. Created OOZIE-1341
> > > >>> >>
> > > >>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com>
> > wrote:
> > > >>> >>
> > > >>> >> >I think I follow what you're saying; assuming everything works
> > > >>> correctly,
> > > >>> >> >I
> > > >>> >> >think that sounds good.  Can you make a JIRA with that and put
> > up a
> > > >>> >> patch;
> > > >>> >> >that might be easier to see how this would work.
> > > >>> >> >
> > > >>> >> >thanks
> > > >>> >> >- Robert
> > > >>> >> >
> > > >>> >> >
> > > >>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <
> > > virag@yahoo-inc.com
> > > >>> >
> > > >>> >> >wrote:
> > > >>> >> >
> > > >>> >> >> We can keep the Main classes in the share lib, but have the
> > > action
> > > >>> >> >> executors package the main classes for shipping it to
> > > >>> >> >> Launcher (basically again having the method
> > getLauncherClasses()
> > > in
> > > >>> >> >>action
> > > >>> >> >> executors). Also the oozie-sharelib.jar containing main
> classes
> > > >>> >> >> should be part of web-app.
> > > >>> >> >> This will improve the building/testing env as classes are in
> > > >>> isolated
> > > >>> >> >> sharelibs but will make the share lib optional as main
> classes
> > > are
> > > >>> part
> > > >>> >> >>of
> > > >>> >> >> oozie-server.
> > > >>> >> >> Can we do this change to trunk as this is blocking our QE's
> to
> > > pick
> > > >>> >> >>builds
> > > >>> >> >> due to share lib incompatibility?
> > > >>> >> >>
> > > >>> >> >> We can still do what oozie-1318 is trying to achieve. For,
> e.g
> > > with
> > > >>> >> Hcat
> > > >>> >> >> integration,
> > > >>> >> >> there can be multiple overridable implementation of
> URIHandlers
> > > >>> that
> > > >>> >> can
> > > >>> >> >> be shipped to launcher.
> > > >>> >> >> We can do the same for other main classes if required.
> > > >>> >> >>
> > > >>> >> >>
> > > >>> >> >> Thanks,
> > > >>> >> >> Virag
> > > >>> >> >>
> > > >>> >> >>
> > > >>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com>
> > > wrote:
> > > >>> >> >>
> > > >>> >> >> >That's true, its primarily a building/testing improvement.
> >  And
> > > >>> from a
> > > >>> >> >> >design perspective, it could be considered "cleaner" to not
> > have
> > > >>> the
> > > >>> >> >> >launcher jar.
> > > >>> >> >> >
> > > >>> >> >> >Your idea sounds like a good compromise between backwards
> > > >>> >> compatibility
> > > >>> >> >> >and
> > > >>> >> >> >improving the building/testing env.  How would we do that?
> > > >>> >> >> >
> > > >>> >> >> >As a side note, there's also a new feature with the Main
> > classes
> > > >>> being
> > > >>> >> >>in
> > > >>> >> >> >the sharelib; users can now override the Main class that
> Oozie
> > > >>> will
> > > >>> >> use
> > > >>> >> >> >for
> > > >>> >> >> >the action because its no longer in the launcher jar.  I
> think
> > > >>> this
> > > >>> >> >>could
> > > >>> >> >> >still be a useful feature, but I'm guessing that if we do
> the
> > > >>> >> >>compromise,
> > > >>> >> >> >this would go away?  In any case, OOZIE-1318 is to make the
> > Main
> > > >>> >> >> >overridable with a config, which is probably cleaner and
> more
> > > >>> flexible
> > > >>> >> >> >than
> > > >>> >> >> >replacing a jar file.
> > > >>> >> >> >
> > > >>> >> >> >thanks
> > > >>> >> >> >- Robert
> > > >>> >> >> >
> > > >>> >> >> >
> > > >>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <
> > > >>> virag@yahoo-inc.com>
> > > >>> >> >> >wrote:
> > > >>> >> >> >
> > > >>> >> >> >> That is a good suggestion for preserving compatibility,
> > > Robert.
> > > >>> >> >> >>
> > > >>> >> >> >> Another question is what is the main purpose of
> OOZIE-1311?
> > > From
> > > >>> >> JIRA
> > > >>> >> >> >> description, the primary
> > > >>> >> >> >> advantage of moving Main classes from core to share lib is
> > it
> > > >>> can
> > > >>> >> >>help
> > > >>> >> >> >>in
> > > >>> >> >> >> preventing dependencies conflicts
> > > >>> >> >> >> (different versions of antlr's for pig, hive). However,
> that
> > > >>> problem
> > > >>> >> >> >>only
> > > >>> >> >> >> exists while building/testing Oozie.
> > > >>> >> >> >> If that is the case, main classes can be moved to share
> lib
> > > but
> > > >>> they
> > > >>> >> >>can
> > > >>> >> >> >> be bundled in oozie-server itself.
> > > >>> >> >> >> In that case, the deployment doesn't change and share lib
> > can
> > > >>> still
> > > >>> >> >>be
> > > >>> >> >> >> optional
> > > >>> >> >> >> as oozie server would ship those classes (same as what was
> > > >>> happening
> > > >>> >> >> >> before). And,
> > > >>> >> >> >> advantage of 1311 for building/testing will be retained
> > > >>> >> >> >>
> > > >>> >> >> >> The main concern is having share lib as a required
> > > installation
> > > >>> will
> > > >>> >> >> >> complicate deployment.
> > > >>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie
> > > doesn't
> > > >>> >> >>provide
> > > >>> >> >> >> inbuilt support for that.
> > > >>> >> >> >> Even though end-users wouldn't notice the change, it will
> > be a
> > > >>> big
> > > >>> >> >> >>change
> > > >>> >> >> >> for system admins installing oozie
> > > >>> >> >> >> So even though share lib is nice feature to have, keeping
> it
> > > >>> >> optional
> > > >>> >> >> >> seems better to me as there are other ways
> > > >>> >> >> >> Of including jars and which were introduced much before
> > share
> > > >>> lib.
> > > >>> >> >> >>
> > > >>> >> >> >> Thanks,
> > > >>> >> >> >> Virag
> > > >>> >> >> >>
> > > >>> >> >> >>
> > > >>> >> >> >>
> > > >>> >> >> >>
> > > >>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rkanter@cloudera.com
> >
> > > >>> wrote:
> > > >>> >> >> >>
> > > >>> >> >> >> >Alejandro had another suggestion to prevent breaking
> stuff
> > > for
> > > >>> >> >>existing
> > > >>> >> >> >> >apps where the sharelib isn't used.  Assuming the
> sharelib
> > is
> > > >>> >> >> >>installed,
> > > >>> >> >> >> >if
> > > >>> >> >> >> >the user doesn't set oozie.use.system.libpath to true,
> then
> > > >>> Oozie
> > > >>> >> >>would
> > > >>> >> >> >> >only include the oozie-*.jar from the sharelib (the one
> > with
> > > >>> the
> > > >>> >> >>Main)
> > > >>> >> >> >>in
> > > >>> >> >> >> >it; if its set to true, then Oozie would include all of
> the
> > > >>> jars in
> > > >>> >> >>the
> > > >>> >> >> >> >sharelib.  This would allow users to still use their own
> > lib
> > > >>> dir
> > > >>> >> >>while
> > > >>> >> >> >> >still getting the Main into the classpath.
> > > >>> >> >> >> >
> > > >>> >> >> >> >- Robert
> > > >>> >> >> >> >
> > > >>> >> >> >> >
> > > >>> >> >> >> >
> > > >>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
> > > >>> >> >> >> ><tu...@gmail.com>wrote:
> > > >>> >> >> >> >
> > > >>> >> >> >> >> virag, if your oozie servers dont have sharelibs
> > currently
> > > >>> >> >>installed
> > > >>> >> >> >> >>then
> > > >>> >> >> >> >> unstalling sharelibs as part of an update would not
> break
> > > ant
> > > >>> >> >>running
> > > >>> >> >> >> >>app.
> > > >>> >> >> >> >> regarding the requirement of enabling the sharelibs to
> > have
> > > >>> the
> > > >>> >> >> >>mains,
> > > >>> >> >> >> >>you
> > > >>> >> >> >> >> have a point. we need to figure out how to get those in
> > the
> > > >>> >> >> >>classpath if
> > > >>> >> >> >> >> the user is not using the sharelibs. i'd suggest we
> leave
> > > >>> this
> > > >>> >> >> >> >> patch in trunk for now and we move it to a release
> branch
> > > >>> once we
> > > >>> >> >> >>sorted
> > > >>> >> >> >> >> out a way of solving this incompatibility.
> > > >>> >> >> >> >>
> > > >>> >> >> >> >> thanks
> > > >>> >> >> >> >>
> > > >>> >> >> >> >> Alejandro
> > > >>> >> >> >> >> (phone typing)
> > > >>> >> >> >> >>
> > > >>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <
> > > >>> virag@yahoo-inc.com>
> > > >>> >> >> >>wrote:
> > > >>> >> >> >> >>
> > > >>> >> >> >> >> >
> > > >>> >> >> >> >> > Tucu, Thanks for the reply and explanation.
> > > >>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a
> > > feasible
> > > >>> >> >>option
> > > >>> >> >> >>as
> > > >>> >> >> >> >>we
> > > >>> >> >> >> >> > cannot expect all the running actions to complete
> > before
> > > >>> Oozie
> > > >>> >> >> >> >>restart.
> > > >>> >> >> >> >> > Hot upgrade is a good option, but we will need time
> to
> > > >>> >> >>implement it
> > > >>> >> >> >> >>as we
> > > >>> >> >> >> >> > currently don't have sharelib as part of our
> > deployment.
> > > >>> >> >> >> >> > So can we deprecate the refactoring of launcher
> classes
> > > in
> > > >>> this
> > > >>> >> >> >> >>release?
> > > >>> >> >> >> >> > Also, all the launcher classes are only added to
> > > >>> distributed
> > > >>> >> >>cache
> > > >>> >> >> >>if
> > > >>> >> >> >> >>the
> > > >>> >> >> >> >> > workflow is configured to use system lib path
> > > >>> >> >> >> >> > (oozie.use.system.libpath=true)
> > > >>> >> >> >> >> > We shouldn't mandate the workflow to have this
> property
> > > as
> > > >>> they
> > > >>> >> >>are
> > > >>> >> >> >> >> > oozie's launcher classes.
> > > >>> >> >> >> >> >
> > > >>> >> >> >> >> > Thanks,
> > > >>> >> >> >> >> > Virag
> > > >>> >> >> >> >> >
> > > >>> >> >> >> >> >
> > > >>> >> >> >> >> >
> > > >>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <
> > > tucu00@gmail.com
> > > >>> >
> > > >>> >> >>wrote:
> > > >>> >> >> >> >> >
> > > >>> >> >> >> >> >> Virag,
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> On sharelib being required, yes you are correct. For
> > > this
> > > >>> we
> > > >>> >> >> >>should:
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs
> that
> > > the
> > > >>> >> >> >>sharelib
> > > >>> >> >> >> >>is
> > > >>> >> >> >> >> >> REQUIRED.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> * Add a check at Oozie startup to verify the
> sharelib
> > > dir
> > > >>> >> >>exists,
> > > >>> >> >> >> >>else
> > > >>> >> >> >> >> >> fail
> > > >>> >> >> >> >> >> to start.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight
> > jobs.
> > > >>> >> >> >>Depending on
> > > >>> >> >> >> >> the
> > > >>> >> >> >> >> >> type of upgrade this may be an issue.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that does
> > not
> > > >>> >> change
> > > >>> >> >> >>the
> > > >>> >> >> >> >> >> sharelib files, this is not an issue and you can
> just
> > > >>> shutdown
> > > >>> >> >>the
> > > >>> >> >> >> >>oozie
> > > >>> >> >> >> >> >> server with in-flight jobs.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that
> > involves
> > > >>> >> >>sharelib
> > > >>> >> >> >> >>files
> > > >>> >> >> >> >> >> changes, then you have 2 options:
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to
> > > suspend/completion,
> > > >>> wait
> > > >>> >> >> >>till
> > > >>> >> >> >> >>all
> > > >>> >> >> >> >> >> running actions end, then shutdown oozie server,
> > upgrade
> > > >>> oozie
> > > >>> >> >> >>server
> > > >>> >> >> >> >> and
> > > >>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF
> > jobs.
> > > In
> > > >>> >> this
> > > >>> >> >> >>case
> > > >>> >> >> >> >>all
> > > >>> >> >> >> >> >> new WF actions will use the new sharelib.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the
> > > >>> oozie-site.xml
> > > >>> >> >> >>sharelib
> > > >>> >> >> >> >> >> location to point to a new directory. upgrade the
> > oozie
> > > >>> >> server.
> > > >>> >> >> >> >>install
> > > >>> >> >> >> >> >> the
> > > >>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in
> HDFS
> > > >>> does
> > > >>> >> not
> > > >>> >> >> >> >>exist).
> > > >>> >> >> >> >> >> start the oozie server. In this case all running WF
> > > >>> actions
> > > >>> >> >>will
> > > >>> >> >> >> >> continue
> > > >>> >> >> >> >> >> running with no issues as the JARs in the
> distributed
> > > >>> cache
> > > >>> >> >>have
> > > >>> >> >> >>not
> > > >>> >> >> >> >> been
> > > >>> >> >> >> >> >> touch. All new WF actions will start using the new
> > > >>> sharelib.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> Note that this sharelib upgrade protocol is not
> > > >>> introduced by
> > > >>> >> >> >> >>requiring
> > > >>> >> >> >> >> >> sharelib, it is required if you have applications
> that
> > > use
> > > >>> >> >> >>sharelib
> > > >>> >> >> >> >> today.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> Does this address your concerns?
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> Thanks.
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
> > > >>> >> >> >><vi...@yahoo-inc.com>
> > > >>> >> >> >> >> >> wrote:
> > > >>> >> >> >> >> >>
> > > >>> >> >> >> >> >>> Hi,
> > > >>> >> >> >> >> >>>
> > > >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems
> to
> > > >>> move all
> > > >>> >> >>the
> > > >>> >> >> >> >> >>> launcher
> > > >>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to  their
> > > respective
> > > >>> >> >> >>sharelibs.
> > > >>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step.
> > > Before
> > > >>> >> >>shared
> > > >>> >> >> >>lib
> > > >>> >> >> >> >> was
> > > >>> >> >> >> >> >>> optional as users could bundle jars with their
> > workflow
> > > >>> >> >> >>application.
> > > >>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2
> > > >>> problems:
> > > >>> >> >> >> >> >>>
> > > >>> >> >> >> >> >>>  1.  The current deployments which don't use action
> > > >>> shared
> > > >>> >> lib
> > > >>> >> >> >>will
> > > >>> >> >> >> >> >>> fail.
> > > >>> >> >> >> >> >>> So, probably we should deprecate the current
> > behavior.
> > > >>> >> >> >> >> >>>
> > > >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will
> fail a
> > > >>> job if
> > > >>> >> >>the
> > > >>> >> >> >> >>files
> > > >>> >> >> >> >> >>> in
> > > >>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is
> > running.
> > > >>> So,
> > > >>> >> >>when
> > > >>> >> >> >> >>Oozie
> > > >>> >> >> >> >> >>> is
> > > >>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as
> part
> > of
> > > >>> >> >> >> >> >>>             deployment, hadoop  will fail the
> > existing
> > > >>> jobs
> > > >>> >> >>for
> > > >>> >> >> >> >>which
> > > >>> >> >> >> >> >>> the
> > > >>> >> >> >> >> >>> timestamp of  the file on hdfs doesn't match the
> > > >>> timestamp of
> > > >>> >> >>its
> > > >>> >> >> >> >>copy
> > > >>> >> >> >> >> >>> in
> > > >>> >> >> >> >> >>> the job's DC.
> > > >>> >> >> >> >> >>>
> > > >>> >> >> >> >> >>>
> > > >>> >> >> >> >> >>> Thanks,
> > > >>> >> >> >> >> >>> Virag
> > > >>> >> >> >> >> >
> > > >>> >> >> >> >>
> > > >>> >> >> >>
> > > >>> >> >> >>
> > > >>> >> >>
> > > >>> >> >>
> > > >>> >>
> > > >>> >>
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > > >
> > > > --
> > > > Alejandro
> > > >
> > >
> >
>

Re: OOZIE-1311

[Rohini Palaniswamy <ro...@gmail.com>]

Robert,
   Do you want to spend effort to rebase and put this patch in branch-4
also? I was wondering since we were thinking of releasing trunk with SLA
work along with HCat as 4.0 instead of releasing the current branch-4 to
consolidate the database changes. If you would like to release branch-4,
then we need to do the OozieDBCLI table changes for SLA work as 5.0 and
change trunk to 5.0 then.


Regards,
Rohini


On Thu, Jun 6, 2013 at 10:04 AM, Robert Kanter <rk...@cloudera.com> wrote:

> It took forever, but I finally got the refactoring done; I just uploaded a
> patch at OOZIE-1315 <https://issues.apache.org/jira/browse/OOZIE-1315>.
>  Please take a look.
>
> The patch has the default set to true (use the launcher jar).  When I
> commit it I can have the version in branch-4 use true and the version in
> trunk use false.
>
> thanks
> - Robert
>
>
> On Mon, May 6, 2013 at 3:06 PM, Rohini Palaniswamy
> <ro...@gmail.com>wrote:
>
> > Tucu,
> >    That sounds good. Can we have the changes go in trunk first and once
> we
> > validate that everything is fine, can we then port all the patches to
> > branch-4.0?
> >
> > Thanks,
> > Rohini
> >
> >
> > On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> >
> > > Rohini,
> > >
> > > I understand you don't want to destabilize Oozie 4. Still, as we have
> no
> > > done a 4 release yet, I think it is a good time to introduce this
> > feature.
> > >
> > > How about the following (similar to what we did for command
> > interruptions)?
> > >
> > > We do as Robert proposed in his last email, but keeping the default
> value
> > > on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON.
> > >
> > > This would mean you don't have to change anything in your deployment
> > > protocol for Oozie 4.
> > >
> > > Thx
> > >
> > >
> > >
> > > On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy <
> > > rohini.aditya@gmail.com> wrote:
> > >
> > >> Robert,
> > >> > A new major release, like Oozie 4, would be a good time to introduce
> > >> this change as the default;
> > >>
> > >>    We already have Oozie 4 deployed in some clusters and it is stable
> > >> now. We are waiting for users to validate Oozie-HCat integration
> before
> > we
> > >> deploy them in production clusters and call out a release in Apache.
> So
> > it
> > >> would be good to introduce this in the next release which will also
> be a
> > >> big release anyway.
> > >>
> > >> Regards,
> > >> Rohini
> > >>
> > >>
> > >> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkanter@cloudera.com
> > >wrote:
> > >>
> > >>> Sorry, I realize that I wasn't clear on that point.  The Main classes
> > >>> would
> > >>> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/)
> > but
> > >>> the other few classes that all actions use in the launcher jar (e.g.
> > >>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie),
> which
> > >>> IIRC is already included when you use sharelibs (for example, when
> you
> > >>> use
> > >>> the sharelib with the Pig action, it includes both the share/lib/pig
> > and
> > >>> share/lib/oozie).
> > >>>
> > >>> thanks
> > >>> - Robert
> > >>>
> > >>>
> > >>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com>
> > >>> wrote:
> > >>>
> > >>> >  Hi Robert,
> > >>> >
> > >>> >  I didn't understand completely. Are you suggesting to have all
> > action
> > >>> > specific main classes in /share/lib/oozie instead of their
> respective
> > >>> > action sharelibs?
> > >>> >
> > >>> >  Thanks,
> > >>> > Virag
> > >>> >
> > >>> >   From: Robert Kanter <rk...@cloudera.com>
> > >>> > Date: Monday, May 6, 2013 12:47 PM
> > >>> > To: "dev@oozie.apache.org" <de...@oozie.apache.org>, Virag Kothari <
> > >>> > virag@yahoo-inc.com>, Alejandro Abdelnur <tu...@cloudera.com>
> > >>> > Subject: Re: OOZIE-1311
> > >>> >
> > >>> >   Virag,
> > >>> > I talked more with Alejandro about the sharelib and the launcher
> jar.
> > >>> >
> > >>> > It would be good if we could eventually get rid of the launcher jar
> > >>> > completely because:
> > >>> >  - less files would be copied to HDFS so it should be faster
> > >>> > - it would fix an issue where some linux distress cleanup the temp
> > dir
> > >>> and
> > >>> > delete the launcher jar
> > >>> >
> > >>> >  The contents of the launcher jar could be moved to the Oozie
> > sharelib
> > >>> > (/share/lib/oozie) which IIRC currently only has a json jar file.
> > >>> >
> > >>> >  As for backwards compatibility, we could do something like what
> you
> > >>> > suggested for the other sharelibs where the webapp module includes
> > >>> them.
> > >>> >  Then we'd add some property to oozie-site that would switch
> between
> > >>> the
> > >>> > old behavior with the launcher jar (sharelib is not required) and
> the
> > >>> new
> > >>> > behavior without the launcher jar (sharelib is required).  A new
> > major
> > >>> > release, like Oozie 4, would be a good time to introduce this
> change
> > >>> as the
> > >>> > default; users can use the property to go back to the old behavior.
> > >>>  Also,
> > >>> > if set to use the new behavior, we should make
> > >>> oozie.use.system.libpath in
> > >>> > the job.properties default to true because the sharelib would be
> > >>> required,
> > >>> > and if set to the old behavior, it would default to false.
> > >>> >
> > >>> >  Thoughts?
> > >>> >
> > >>> >  thanks
> > >>> > - Robert
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <
> virag@yahoo-inc.com
> > >>> >wrote:
> > >>> >
> > >>> >> Thanks Robert. Created OOZIE-1341
> > >>> >>
> > >>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com>
> wrote:
> > >>> >>
> > >>> >> >I think I follow what you're saying; assuming everything works
> > >>> correctly,
> > >>> >> >I
> > >>> >> >think that sounds good.  Can you make a JIRA with that and put
> up a
> > >>> >> patch;
> > >>> >> >that might be easier to see how this would work.
> > >>> >> >
> > >>> >> >thanks
> > >>> >> >- Robert
> > >>> >> >
> > >>> >> >
> > >>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <
> > virag@yahoo-inc.com
> > >>> >
> > >>> >> >wrote:
> > >>> >> >
> > >>> >> >> We can keep the Main classes in the share lib, but have the
> > action
> > >>> >> >> executors package the main classes for shipping it to
> > >>> >> >> Launcher (basically again having the method
> getLauncherClasses()
> > in
> > >>> >> >>action
> > >>> >> >> executors). Also the oozie-sharelib.jar containing main classes
> > >>> >> >> should be part of web-app.
> > >>> >> >> This will improve the building/testing env as classes are in
> > >>> isolated
> > >>> >> >> sharelibs but will make the share lib optional as main classes
> > are
> > >>> part
> > >>> >> >>of
> > >>> >> >> oozie-server.
> > >>> >> >> Can we do this change to trunk as this is blocking our QE's to
> > pick
> > >>> >> >>builds
> > >>> >> >> due to share lib incompatibility?
> > >>> >> >>
> > >>> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g
> > with
> > >>> >> Hcat
> > >>> >> >> integration,
> > >>> >> >> there can be multiple overridable implementation of URIHandlers
> > >>> that
> > >>> >> can
> > >>> >> >> be shipped to launcher.
> > >>> >> >> We can do the same for other main classes if required.
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> Thanks,
> > >>> >> >> Virag
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com>
> > wrote:
> > >>> >> >>
> > >>> >> >> >That's true, its primarily a building/testing improvement.
>  And
> > >>> from a
> > >>> >> >> >design perspective, it could be considered "cleaner" to not
> have
> > >>> the
> > >>> >> >> >launcher jar.
> > >>> >> >> >
> > >>> >> >> >Your idea sounds like a good compromise between backwards
> > >>> >> compatibility
> > >>> >> >> >and
> > >>> >> >> >improving the building/testing env.  How would we do that?
> > >>> >> >> >
> > >>> >> >> >As a side note, there's also a new feature with the Main
> classes
> > >>> being
> > >>> >> >>in
> > >>> >> >> >the sharelib; users can now override the Main class that Oozie
> > >>> will
> > >>> >> use
> > >>> >> >> >for
> > >>> >> >> >the action because its no longer in the launcher jar.  I think
> > >>> this
> > >>> >> >>could
> > >>> >> >> >still be a useful feature, but I'm guessing that if we do the
> > >>> >> >>compromise,
> > >>> >> >> >this would go away?  In any case, OOZIE-1318 is to make the
> Main
> > >>> >> >> >overridable with a config, which is probably cleaner and more
> > >>> flexible
> > >>> >> >> >than
> > >>> >> >> >replacing a jar file.
> > >>> >> >> >
> > >>> >> >> >thanks
> > >>> >> >> >- Robert
> > >>> >> >> >
> > >>> >> >> >
> > >>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <
> > >>> virag@yahoo-inc.com>
> > >>> >> >> >wrote:
> > >>> >> >> >
> > >>> >> >> >> That is a good suggestion for preserving compatibility,
> > Robert.
> > >>> >> >> >>
> > >>> >> >> >> Another question is what is the main purpose of OOZIE-1311?
> > From
> > >>> >> JIRA
> > >>> >> >> >> description, the primary
> > >>> >> >> >> advantage of moving Main classes from core to share lib is
> it
> > >>> can
> > >>> >> >>help
> > >>> >> >> >>in
> > >>> >> >> >> preventing dependencies conflicts
> > >>> >> >> >> (different versions of antlr's for pig, hive). However, that
> > >>> problem
> > >>> >> >> >>only
> > >>> >> >> >> exists while building/testing Oozie.
> > >>> >> >> >> If that is the case, main classes can be moved to share lib
> > but
> > >>> they
> > >>> >> >>can
> > >>> >> >> >> be bundled in oozie-server itself.
> > >>> >> >> >> In that case, the deployment doesn't change and share lib
> can
> > >>> still
> > >>> >> >>be
> > >>> >> >> >> optional
> > >>> >> >> >> as oozie server would ship those classes (same as what was
> > >>> happening
> > >>> >> >> >> before). And,
> > >>> >> >> >> advantage of 1311 for building/testing will be retained
> > >>> >> >> >>
> > >>> >> >> >> The main concern is having share lib as a required
> > installation
> > >>> will
> > >>> >> >> >> complicate deployment.
> > >>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie
> > doesn't
> > >>> >> >>provide
> > >>> >> >> >> inbuilt support for that.
> > >>> >> >> >> Even though end-users wouldn't notice the change, it will
> be a
> > >>> big
> > >>> >> >> >>change
> > >>> >> >> >> for system admins installing oozie
> > >>> >> >> >> So even though share lib is nice feature to have, keeping it
> > >>> >> optional
> > >>> >> >> >> seems better to me as there are other ways
> > >>> >> >> >> Of including jars and which were introduced much before
> share
> > >>> lib.
> > >>> >> >> >>
> > >>> >> >> >> Thanks,
> > >>> >> >> >> Virag
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com>
> > >>> wrote:
> > >>> >> >> >>
> > >>> >> >> >> >Alejandro had another suggestion to prevent breaking stuff
> > for
> > >>> >> >>existing
> > >>> >> >> >> >apps where the sharelib isn't used.  Assuming the sharelib
> is
> > >>> >> >> >>installed,
> > >>> >> >> >> >if
> > >>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then
> > >>> Oozie
> > >>> >> >>would
> > >>> >> >> >> >only include the oozie-*.jar from the sharelib (the one
> with
> > >>> the
> > >>> >> >>Main)
> > >>> >> >> >>in
> > >>> >> >> >> >it; if its set to true, then Oozie would include all of the
> > >>> jars in
> > >>> >> >>the
> > >>> >> >> >> >sharelib.  This would allow users to still use their own
> lib
> > >>> dir
> > >>> >> >>while
> > >>> >> >> >> >still getting the Main into the classpath.
> > >>> >> >> >> >
> > >>> >> >> >> >- Robert
> > >>> >> >> >> >
> > >>> >> >> >> >
> > >>> >> >> >> >
> > >>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
> > >>> >> >> >> ><tu...@gmail.com>wrote:
> > >>> >> >> >> >
> > >>> >> >> >> >> virag, if your oozie servers dont have sharelibs
> currently
> > >>> >> >>installed
> > >>> >> >> >> >>then
> > >>> >> >> >> >> unstalling sharelibs as part of an update would not break
> > ant
> > >>> >> >>running
> > >>> >> >> >> >>app.
> > >>> >> >> >> >> regarding the requirement of enabling the sharelibs to
> have
> > >>> the
> > >>> >> >> >>mains,
> > >>> >> >> >> >>you
> > >>> >> >> >> >> have a point. we need to figure out how to get those in
> the
> > >>> >> >> >>classpath if
> > >>> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave
> > >>> this
> > >>> >> >> >> >> patch in trunk for now and we move it to a release branch
> > >>> once we
> > >>> >> >> >>sorted
> > >>> >> >> >> >> out a way of solving this incompatibility.
> > >>> >> >> >> >>
> > >>> >> >> >> >> thanks
> > >>> >> >> >> >>
> > >>> >> >> >> >> Alejandro
> > >>> >> >> >> >> (phone typing)
> > >>> >> >> >> >>
> > >>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <
> > >>> virag@yahoo-inc.com>
> > >>> >> >> >>wrote:
> > >>> >> >> >> >>
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Tucu, Thanks for the reply and explanation.
> > >>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a
> > feasible
> > >>> >> >>option
> > >>> >> >> >>as
> > >>> >> >> >> >>we
> > >>> >> >> >> >> > cannot expect all the running actions to complete
> before
> > >>> Oozie
> > >>> >> >> >> >>restart.
> > >>> >> >> >> >> > Hot upgrade is a good option, but we will need time to
> > >>> >> >>implement it
> > >>> >> >> >> >>as we
> > >>> >> >> >> >> > currently don't have sharelib as part of our
> deployment.
> > >>> >> >> >> >> > So can we deprecate the refactoring of launcher classes
> > in
> > >>> this
> > >>> >> >> >> >>release?
> > >>> >> >> >> >> > Also, all the launcher classes are only added to
> > >>> distributed
> > >>> >> >>cache
> > >>> >> >> >>if
> > >>> >> >> >> >>the
> > >>> >> >> >> >> > workflow is configured to use system lib path
> > >>> >> >> >> >> > (oozie.use.system.libpath=true)
> > >>> >> >> >> >> > We shouldn't mandate the workflow to have this property
> > as
> > >>> they
> > >>> >> >>are
> > >>> >> >> >> >> > oozie's launcher classes.
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > Thanks,
> > >>> >> >> >> >> > Virag
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >
> > >>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <
> > tucu00@gmail.com
> > >>> >
> > >>> >> >>wrote:
> > >>> >> >> >> >> >
> > >>> >> >> >> >> >> Virag,
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> On sharelib being required, yes you are correct. For
> > this
> > >>> we
> > >>> >> >> >>should:
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that
> > the
> > >>> >> >> >>sharelib
> > >>> >> >> >> >>is
> > >>> >> >> >> >> >> REQUIRED.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib
> > dir
> > >>> >> >>exists,
> > >>> >> >> >> >>else
> > >>> >> >> >> >> >> fail
> > >>> >> >> >> >> >> to start.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight
> jobs.
> > >>> >> >> >>Depending on
> > >>> >> >> >> >> the
> > >>> >> >> >> >> >> type of upgrade this may be an issue.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that does
> not
> > >>> >> change
> > >>> >> >> >>the
> > >>> >> >> >> >> >> sharelib files, this is not an issue and you can just
> > >>> shutdown
> > >>> >> >>the
> > >>> >> >> >> >>oozie
> > >>> >> >> >> >> >> server with in-flight jobs.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> * If you are upgrading an oozie server fix that
> involves
> > >>> >> >>sharelib
> > >>> >> >> >> >>files
> > >>> >> >> >> >> >> changes, then you have 2 options:
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to
> > suspend/completion,
> > >>> wait
> > >>> >> >> >>till
> > >>> >> >> >> >>all
> > >>> >> >> >> >> >> running actions end, then shutdown oozie server,
> upgrade
> > >>> oozie
> > >>> >> >> >>server
> > >>> >> >> >> >> and
> > >>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF
> jobs.
> > In
> > >>> >> this
> > >>> >> >> >>case
> > >>> >> >> >> >>all
> > >>> >> >> >> >> >> new WF actions will use the new sharelib.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the
> > >>> oozie-site.xml
> > >>> >> >> >>sharelib
> > >>> >> >> >> >> >> location to point to a new directory. upgrade the
> oozie
> > >>> >> server.
> > >>> >> >> >> >>install
> > >>> >> >> >> >> >> the
> > >>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS
> > >>> does
> > >>> >> not
> > >>> >> >> >> >>exist).
> > >>> >> >> >> >> >> start the oozie server. In this case all running WF
> > >>> actions
> > >>> >> >>will
> > >>> >> >> >> >> continue
> > >>> >> >> >> >> >> running with no issues as the JARs in the distributed
> > >>> cache
> > >>> >> >>have
> > >>> >> >> >>not
> > >>> >> >> >> >> been
> > >>> >> >> >> >> >> touch. All new WF actions will start using the new
> > >>> sharelib.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Note that this sharelib upgrade protocol is not
> > >>> introduced by
> > >>> >> >> >> >>requiring
> > >>> >> >> >> >> >> sharelib, it is required if you have applications that
> > use
> > >>> >> >> >>sharelib
> > >>> >> >> >> >> today.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Does this address your concerns?
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> Thanks.
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
> > >>> >> >> >><vi...@yahoo-inc.com>
> > >>> >> >> >> >> >> wrote:
> > >>> >> >> >> >> >>
> > >>> >> >> >> >> >>> Hi,
> > >>> >> >> >> >> >>>
> > >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to
> > >>> move all
> > >>> >> >>the
> > >>> >> >> >> >> >>> launcher
> > >>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to  their
> > respective
> > >>> >> >> >>sharelibs.
> > >>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step.
> > Before
> > >>> >> >>shared
> > >>> >> >> >>lib
> > >>> >> >> >> >> was
> > >>> >> >> >> >> >>> optional as users could bundle jars with their
> workflow
> > >>> >> >> >>application.
> > >>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2
> > >>> problems:
> > >>> >> >> >> >> >>>
> > >>> >> >> >> >> >>>  1.  The current deployments which don't use action
> > >>> shared
> > >>> >> lib
> > >>> >> >> >>will
> > >>> >> >> >> >> >>> fail.
> > >>> >> >> >> >> >>> So, probably we should deprecate the current
> behavior.
> > >>> >> >> >> >> >>>
> > >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a
> > >>> job if
> > >>> >> >>the
> > >>> >> >> >> >>files
> > >>> >> >> >> >> >>> in
> > >>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is
> running.
> > >>> So,
> > >>> >> >>when
> > >>> >> >> >> >>Oozie
> > >>> >> >> >> >> >>> is
> > >>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part
> of
> > >>> >> >> >> >> >>>             deployment, hadoop  will fail the
> existing
> > >>> jobs
> > >>> >> >>for
> > >>> >> >> >> >>which
> > >>> >> >> >> >> >>> the
> > >>> >> >> >> >> >>> timestamp of  the file on hdfs doesn't match the
> > >>> timestamp of
> > >>> >> >>its
> > >>> >> >> >> >>copy
> > >>> >> >> >> >> >>> in
> > >>> >> >> >> >> >>> the job's DC.
> > >>> >> >> >> >> >>>
> > >>> >> >> >> >> >>>
> > >>> >> >> >> >> >>> Thanks,
> > >>> >> >> >> >> >>> Virag
> > >>> >> >> >> >> >
> > >>> >> >> >> >>
> > >>> >> >> >>
> > >>> >> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >>
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >>
> > >
> > >
> > > --
> > > Alejandro
> > >
> >
>

Re: OOZIE-1311

[Robert Kanter <rk...@cloudera.com>]

It took forever, but I finally got the refactoring done; I just uploaded a
patch at OOZIE-1315 <https://issues.apache.org/jira/browse/OOZIE-1315>.
 Please take a look.

The patch has the default set to true (use the launcher jar).  When I
commit it I can have the version in branch-4 use true and the version in
trunk use false.

thanks
- Robert


On Mon, May 6, 2013 at 3:06 PM, Rohini Palaniswamy
<ro...@gmail.com>wrote:

> Tucu,
>    That sounds good. Can we have the changes go in trunk first and once we
> validate that everything is fine, can we then port all the patches to
> branch-4.0?
>
> Thanks,
> Rohini
>
>
> On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Rohini,
> >
> > I understand you don't want to destabilize Oozie 4. Still, as we have no
> > done a 4 release yet, I think it is a good time to introduce this
> feature.
> >
> > How about the following (similar to what we did for command
> interruptions)?
> >
> > We do as Robert proposed in his last email, but keeping the default value
> > on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON.
> >
> > This would mean you don't have to change anything in your deployment
> > protocol for Oozie 4.
> >
> > Thx
> >
> >
> >
> > On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy <
> > rohini.aditya@gmail.com> wrote:
> >
> >> Robert,
> >> > A new major release, like Oozie 4, would be a good time to introduce
> >> this change as the default;
> >>
> >>    We already have Oozie 4 deployed in some clusters and it is stable
> >> now. We are waiting for users to validate Oozie-HCat integration before
> we
> >> deploy them in production clusters and call out a release in Apache. So
> it
> >> would be good to introduce this in the next release which will also be a
> >> big release anyway.
> >>
> >> Regards,
> >> Rohini
> >>
> >>
> >> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rkanter@cloudera.com
> >wrote:
> >>
> >>> Sorry, I realize that I wasn't clear on that point.  The Main classes
> >>> would
> >>> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/)
> but
> >>> the other few classes that all actions use in the launcher jar (e.g.
> >>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which
> >>> IIRC is already included when you use sharelibs (for example, when you
> >>> use
> >>> the sharelib with the Pig action, it includes both the share/lib/pig
> and
> >>> share/lib/oozie).
> >>>
> >>> thanks
> >>> - Robert
> >>>
> >>>
> >>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com>
> >>> wrote:
> >>>
> >>> >  Hi Robert,
> >>> >
> >>> >  I didn't understand completely. Are you suggesting to have all
> action
> >>> > specific main classes in /share/lib/oozie instead of their respective
> >>> > action sharelibs?
> >>> >
> >>> >  Thanks,
> >>> > Virag
> >>> >
> >>> >   From: Robert Kanter <rk...@cloudera.com>
> >>> > Date: Monday, May 6, 2013 12:47 PM
> >>> > To: "dev@oozie.apache.org" <de...@oozie.apache.org>, Virag Kothari <
> >>> > virag@yahoo-inc.com>, Alejandro Abdelnur <tu...@cloudera.com>
> >>> > Subject: Re: OOZIE-1311
> >>> >
> >>> >   Virag,
> >>> > I talked more with Alejandro about the sharelib and the launcher jar.
> >>> >
> >>> > It would be good if we could eventually get rid of the launcher jar
> >>> > completely because:
> >>> >  - less files would be copied to HDFS so it should be faster
> >>> > - it would fix an issue where some linux distress cleanup the temp
> dir
> >>> and
> >>> > delete the launcher jar
> >>> >
> >>> >  The contents of the launcher jar could be moved to the Oozie
> sharelib
> >>> > (/share/lib/oozie) which IIRC currently only has a json jar file.
> >>> >
> >>> >  As for backwards compatibility, we could do something like what you
> >>> > suggested for the other sharelibs where the webapp module includes
> >>> them.
> >>> >  Then we'd add some property to oozie-site that would switch between
> >>> the
> >>> > old behavior with the launcher jar (sharelib is not required) and the
> >>> new
> >>> > behavior without the launcher jar (sharelib is required).  A new
> major
> >>> > release, like Oozie 4, would be a good time to introduce this change
> >>> as the
> >>> > default; users can use the property to go back to the old behavior.
> >>>  Also,
> >>> > if set to use the new behavior, we should make
> >>> oozie.use.system.libpath in
> >>> > the job.properties default to true because the sharelib would be
> >>> required,
> >>> > and if set to the old behavior, it would default to false.
> >>> >
> >>> >  Thoughts?
> >>> >
> >>> >  thanks
> >>> > - Robert
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <virag@yahoo-inc.com
> >>> >wrote:
> >>> >
> >>> >> Thanks Robert. Created OOZIE-1341
> >>> >>
> >>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
> >>> >>
> >>> >> >I think I follow what you're saying; assuming everything works
> >>> correctly,
> >>> >> >I
> >>> >> >think that sounds good.  Can you make a JIRA with that and put up a
> >>> >> patch;
> >>> >> >that might be easier to see how this would work.
> >>> >> >
> >>> >> >thanks
> >>> >> >- Robert
> >>> >> >
> >>> >> >
> >>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <
> virag@yahoo-inc.com
> >>> >
> >>> >> >wrote:
> >>> >> >
> >>> >> >> We can keep the Main classes in the share lib, but have the
> action
> >>> >> >> executors package the main classes for shipping it to
> >>> >> >> Launcher (basically again having the method getLauncherClasses()
> in
> >>> >> >>action
> >>> >> >> executors). Also the oozie-sharelib.jar containing main classes
> >>> >> >> should be part of web-app.
> >>> >> >> This will improve the building/testing env as classes are in
> >>> isolated
> >>> >> >> sharelibs but will make the share lib optional as main classes
> are
> >>> part
> >>> >> >>of
> >>> >> >> oozie-server.
> >>> >> >> Can we do this change to trunk as this is blocking our QE's to
> pick
> >>> >> >>builds
> >>> >> >> due to share lib incompatibility?
> >>> >> >>
> >>> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g
> with
> >>> >> Hcat
> >>> >> >> integration,
> >>> >> >> there can be multiple overridable implementation of URIHandlers
> >>> that
> >>> >> can
> >>> >> >> be shipped to launcher.
> >>> >> >> We can do the same for other main classes if required.
> >>> >> >>
> >>> >> >>
> >>> >> >> Thanks,
> >>> >> >> Virag
> >>> >> >>
> >>> >> >>
> >>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com>
> wrote:
> >>> >> >>
> >>> >> >> >That's true, its primarily a building/testing improvement.  And
> >>> from a
> >>> >> >> >design perspective, it could be considered "cleaner" to not have
> >>> the
> >>> >> >> >launcher jar.
> >>> >> >> >
> >>> >> >> >Your idea sounds like a good compromise between backwards
> >>> >> compatibility
> >>> >> >> >and
> >>> >> >> >improving the building/testing env.  How would we do that?
> >>> >> >> >
> >>> >> >> >As a side note, there's also a new feature with the Main classes
> >>> being
> >>> >> >>in
> >>> >> >> >the sharelib; users can now override the Main class that Oozie
> >>> will
> >>> >> use
> >>> >> >> >for
> >>> >> >> >the action because its no longer in the launcher jar.  I think
> >>> this
> >>> >> >>could
> >>> >> >> >still be a useful feature, but I'm guessing that if we do the
> >>> >> >>compromise,
> >>> >> >> >this would go away?  In any case, OOZIE-1318 is to make the Main
> >>> >> >> >overridable with a config, which is probably cleaner and more
> >>> flexible
> >>> >> >> >than
> >>> >> >> >replacing a jar file.
> >>> >> >> >
> >>> >> >> >thanks
> >>> >> >> >- Robert
> >>> >> >> >
> >>> >> >> >
> >>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <
> >>> virag@yahoo-inc.com>
> >>> >> >> >wrote:
> >>> >> >> >
> >>> >> >> >> That is a good suggestion for preserving compatibility,
> Robert.
> >>> >> >> >>
> >>> >> >> >> Another question is what is the main purpose of OOZIE-1311?
> From
> >>> >> JIRA
> >>> >> >> >> description, the primary
> >>> >> >> >> advantage of moving Main classes from core to share lib is it
> >>> can
> >>> >> >>help
> >>> >> >> >>in
> >>> >> >> >> preventing dependencies conflicts
> >>> >> >> >> (different versions of antlr's for pig, hive). However, that
> >>> problem
> >>> >> >> >>only
> >>> >> >> >> exists while building/testing Oozie.
> >>> >> >> >> If that is the case, main classes can be moved to share lib
> but
> >>> they
> >>> >> >>can
> >>> >> >> >> be bundled in oozie-server itself.
> >>> >> >> >> In that case, the deployment doesn't change and share lib can
> >>> still
> >>> >> >>be
> >>> >> >> >> optional
> >>> >> >> >> as oozie server would ship those classes (same as what was
> >>> happening
> >>> >> >> >> before). And,
> >>> >> >> >> advantage of 1311 for building/testing will be retained
> >>> >> >> >>
> >>> >> >> >> The main concern is having share lib as a required
> installation
> >>> will
> >>> >> >> >> complicate deployment.
> >>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie
> doesn't
> >>> >> >>provide
> >>> >> >> >> inbuilt support for that.
> >>> >> >> >> Even though end-users wouldn't notice the change, it will be a
> >>> big
> >>> >> >> >>change
> >>> >> >> >> for system admins installing oozie
> >>> >> >> >> So even though share lib is nice feature to have, keeping it
> >>> >> optional
> >>> >> >> >> seems better to me as there are other ways
> >>> >> >> >> Of including jars and which were introduced much before share
> >>> lib.
> >>> >> >> >>
> >>> >> >> >> Thanks,
> >>> >> >> >> Virag
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com>
> >>> wrote:
> >>> >> >> >>
> >>> >> >> >> >Alejandro had another suggestion to prevent breaking stuff
> for
> >>> >> >>existing
> >>> >> >> >> >apps where the sharelib isn't used.  Assuming the sharelib is
> >>> >> >> >>installed,
> >>> >> >> >> >if
> >>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then
> >>> Oozie
> >>> >> >>would
> >>> >> >> >> >only include the oozie-*.jar from the sharelib (the one with
> >>> the
> >>> >> >>Main)
> >>> >> >> >>in
> >>> >> >> >> >it; if its set to true, then Oozie would include all of the
> >>> jars in
> >>> >> >>the
> >>> >> >> >> >sharelib.  This would allow users to still use their own lib
> >>> dir
> >>> >> >>while
> >>> >> >> >> >still getting the Main into the classpath.
> >>> >> >> >> >
> >>> >> >> >> >- Robert
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> >
> >>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
> >>> >> >> >> ><tu...@gmail.com>wrote:
> >>> >> >> >> >
> >>> >> >> >> >> virag, if your oozie servers dont have sharelibs currently
> >>> >> >>installed
> >>> >> >> >> >>then
> >>> >> >> >> >> unstalling sharelibs as part of an update would not break
> ant
> >>> >> >>running
> >>> >> >> >> >>app.
> >>> >> >> >> >> regarding the requirement of enabling the sharelibs to have
> >>> the
> >>> >> >> >>mains,
> >>> >> >> >> >>you
> >>> >> >> >> >> have a point. we need to figure out how to get those in the
> >>> >> >> >>classpath if
> >>> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave
> >>> this
> >>> >> >> >> >> patch in trunk for now and we move it to a release branch
> >>> once we
> >>> >> >> >>sorted
> >>> >> >> >> >> out a way of solving this incompatibility.
> >>> >> >> >> >>
> >>> >> >> >> >> thanks
> >>> >> >> >> >>
> >>> >> >> >> >> Alejandro
> >>> >> >> >> >> (phone typing)
> >>> >> >> >> >>
> >>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <
> >>> virag@yahoo-inc.com>
> >>> >> >> >>wrote:
> >>> >> >> >> >>
> >>> >> >> >> >> >
> >>> >> >> >> >> > Tucu, Thanks for the reply and explanation.
> >>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a
> feasible
> >>> >> >>option
> >>> >> >> >>as
> >>> >> >> >> >>we
> >>> >> >> >> >> > cannot expect all the running actions to complete before
> >>> Oozie
> >>> >> >> >> >>restart.
> >>> >> >> >> >> > Hot upgrade is a good option, but we will need time to
> >>> >> >>implement it
> >>> >> >> >> >>as we
> >>> >> >> >> >> > currently don't have sharelib as part of our deployment.
> >>> >> >> >> >> > So can we deprecate the refactoring of launcher classes
> in
> >>> this
> >>> >> >> >> >>release?
> >>> >> >> >> >> > Also, all the launcher classes are only added to
> >>> distributed
> >>> >> >>cache
> >>> >> >> >>if
> >>> >> >> >> >>the
> >>> >> >> >> >> > workflow is configured to use system lib path
> >>> >> >> >> >> > (oozie.use.system.libpath=true)
> >>> >> >> >> >> > We shouldn't mandate the workflow to have this property
> as
> >>> they
> >>> >> >>are
> >>> >> >> >> >> > oozie's launcher classes.
> >>> >> >> >> >> >
> >>> >> >> >> >> > Thanks,
> >>> >> >> >> >> > Virag
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> >
> >>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <
> tucu00@gmail.com
> >>> >
> >>> >> >>wrote:
> >>> >> >> >> >> >
> >>> >> >> >> >> >> Virag,
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> On sharelib being required, yes you are correct. For
> this
> >>> we
> >>> >> >> >>should:
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that
> the
> >>> >> >> >>sharelib
> >>> >> >> >> >>is
> >>> >> >> >> >> >> REQUIRED.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib
> dir
> >>> >> >>exists,
> >>> >> >> >> >>else
> >>> >> >> >> >> >> fail
> >>> >> >> >> >> >> to start.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs.
> >>> >> >> >>Depending on
> >>> >> >> >> >> the
> >>> >> >> >> >> >> type of upgrade this may be an issue.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> * If you are upgrading an oozie server fix that does not
> >>> >> change
> >>> >> >> >>the
> >>> >> >> >> >> >> sharelib files, this is not an issue and you can just
> >>> shutdown
> >>> >> >>the
> >>> >> >> >> >>oozie
> >>> >> >> >> >> >> server with in-flight jobs.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> * If you are upgrading an oozie server fix that involves
> >>> >> >>sharelib
> >>> >> >> >> >>files
> >>> >> >> >> >> >> changes, then you have 2 options:
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to
> suspend/completion,
> >>> wait
> >>> >> >> >>till
> >>> >> >> >> >>all
> >>> >> >> >> >> >> running actions end, then shutdown oozie server, upgrade
> >>> oozie
> >>> >> >> >>server
> >>> >> >> >> >> and
> >>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs.
> In
> >>> >> this
> >>> >> >> >>case
> >>> >> >> >> >>all
> >>> >> >> >> >> >> new WF actions will use the new sharelib.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the
> >>> oozie-site.xml
> >>> >> >> >>sharelib
> >>> >> >> >> >> >> location to point to a new directory. upgrade the oozie
> >>> >> server.
> >>> >> >> >> >>install
> >>> >> >> >> >> >> the
> >>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS
> >>> does
> >>> >> not
> >>> >> >> >> >>exist).
> >>> >> >> >> >> >> start the oozie server. In this case all running WF
> >>> actions
> >>> >> >>will
> >>> >> >> >> >> continue
> >>> >> >> >> >> >> running with no issues as the JARs in the distributed
> >>> cache
> >>> >> >>have
> >>> >> >> >>not
> >>> >> >> >> >> been
> >>> >> >> >> >> >> touch. All new WF actions will start using the new
> >>> sharelib.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> Note that this sharelib upgrade protocol is not
> >>> introduced by
> >>> >> >> >> >>requiring
> >>> >> >> >> >> >> sharelib, it is required if you have applications that
> use
> >>> >> >> >>sharelib
> >>> >> >> >> >> today.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> Does this address your concerns?
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> Thanks.
> >>> >> >> >> >> >>
> >>> >> >> >> >> >>
> >>> >> >> >> >> >>
> >>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
> >>> >> >> >><vi...@yahoo-inc.com>
> >>> >> >> >> >> >> wrote:
> >>> >> >> >> >> >>
> >>> >> >> >> >> >>> Hi,
> >>> >> >> >> >> >>>
> >>> >> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to
> >>> move all
> >>> >> >>the
> >>> >> >> >> >> >>> launcher
> >>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to  their
> respective
> >>> >> >> >>sharelibs.
> >>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step.
> Before
> >>> >> >>shared
> >>> >> >> >>lib
> >>> >> >> >> >> was
> >>> >> >> >> >> >>> optional as users could bundle jars with their workflow
> >>> >> >> >>application.
> >>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2
> >>> problems:
> >>> >> >> >> >> >>>
> >>> >> >> >> >> >>>  1.  The current deployments which don't use action
> >>> shared
> >>> >> lib
> >>> >> >> >>will
> >>> >> >> >> >> >>> fail.
> >>> >> >> >> >> >>> So, probably we should deprecate the current behavior.
> >>> >> >> >> >> >>>
> >>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a
> >>> job if
> >>> >> >>the
> >>> >> >> >> >>files
> >>> >> >> >> >> >>> in
> >>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running.
> >>> So,
> >>> >> >>when
> >>> >> >> >> >>Oozie
> >>> >> >> >> >> >>> is
> >>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of
> >>> >> >> >> >> >>>             deployment, hadoop  will fail the existing
> >>> jobs
> >>> >> >>for
> >>> >> >> >> >>which
> >>> >> >> >> >> >>> the
> >>> >> >> >> >> >>> timestamp of  the file on hdfs doesn't match the
> >>> timestamp of
> >>> >> >>its
> >>> >> >> >> >>copy
> >>> >> >> >> >> >>> in
> >>> >> >> >> >> >>> the job's DC.
> >>> >> >> >> >> >>>
> >>> >> >> >> >> >>>
> >>> >> >> >> >> >>> Thanks,
> >>> >> >> >> >> >>> Virag
> >>> >> >> >> >> >
> >>> >> >> >> >>
> >>> >> >> >>
> >>> >> >> >>
> >>> >> >>
> >>> >> >>
> >>> >>
> >>> >>
> >>> >
> >>>
> >>
> >>
> >
> >
> > --
> > Alejandro
> >
>

Re: OOZIE-1311

[Rohini Palaniswamy <ro...@gmail.com>]

Tucu,
   That sounds good. Can we have the changes go in trunk first and once we
validate that everything is fine, can we then port all the patches to
branch-4.0?

Thanks,
Rohini


On Mon, May 6, 2013 at 2:52 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Rohini,
>
> I understand you don't want to destabilize Oozie 4. Still, as we have no
> done a 4 release yet, I think it is a good time to introduce this feature.
>
> How about the following (similar to what we did for command interruptions)?
>
> We do as Robert proposed in his last email, but keeping the default value
> on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON.
>
> This would mean you don't have to change anything in your deployment
> protocol for Oozie 4.
>
> Thx
>
>
>
> On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy <
> rohini.aditya@gmail.com> wrote:
>
>> Robert,
>> > A new major release, like Oozie 4, would be a good time to introduce
>> this change as the default;
>>
>>    We already have Oozie 4 deployed in some clusters and it is stable
>> now. We are waiting for users to validate Oozie-HCat integration before we
>> deploy them in production clusters and call out a release in Apache. So it
>> would be good to introduce this in the next release which will also be a
>> big release anyway.
>>
>> Regards,
>> Rohini
>>
>>
>> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rk...@cloudera.com>wrote:
>>
>>> Sorry, I realize that I wasn't clear on that point.  The Main classes
>>> would
>>> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) but
>>> the other few classes that all actions use in the launcher jar (e.g.
>>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which
>>> IIRC is already included when you use sharelibs (for example, when you
>>> use
>>> the sharelib with the Pig action, it includes both the share/lib/pig and
>>> share/lib/oozie).
>>>
>>> thanks
>>> - Robert
>>>
>>>
>>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com>
>>> wrote:
>>>
>>> >  Hi Robert,
>>> >
>>> >  I didn't understand completely. Are you suggesting to have all action
>>> > specific main classes in /share/lib/oozie instead of their respective
>>> > action sharelibs?
>>> >
>>> >  Thanks,
>>> > Virag
>>> >
>>> >   From: Robert Kanter <rk...@cloudera.com>
>>> > Date: Monday, May 6, 2013 12:47 PM
>>> > To: "dev@oozie.apache.org" <de...@oozie.apache.org>, Virag Kothari <
>>> > virag@yahoo-inc.com>, Alejandro Abdelnur <tu...@cloudera.com>
>>> > Subject: Re: OOZIE-1311
>>> >
>>> >   Virag,
>>> > I talked more with Alejandro about the sharelib and the launcher jar.
>>> >
>>> > It would be good if we could eventually get rid of the launcher jar
>>> > completely because:
>>> >  - less files would be copied to HDFS so it should be faster
>>> > - it would fix an issue where some linux distress cleanup the temp dir
>>> and
>>> > delete the launcher jar
>>> >
>>> >  The contents of the launcher jar could be moved to the Oozie sharelib
>>> > (/share/lib/oozie) which IIRC currently only has a json jar file.
>>> >
>>> >  As for backwards compatibility, we could do something like what you
>>> > suggested for the other sharelibs where the webapp module includes
>>> them.
>>> >  Then we'd add some property to oozie-site that would switch between
>>> the
>>> > old behavior with the launcher jar (sharelib is not required) and the
>>> new
>>> > behavior without the launcher jar (sharelib is required).  A new major
>>> > release, like Oozie 4, would be a good time to introduce this change
>>> as the
>>> > default; users can use the property to go back to the old behavior.
>>>  Also,
>>> > if set to use the new behavior, we should make
>>> oozie.use.system.libpath in
>>> > the job.properties default to true because the sharelib would be
>>> required,
>>> > and if set to the old behavior, it would default to false.
>>> >
>>> >  Thoughts?
>>> >
>>> >  thanks
>>> > - Robert
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <virag@yahoo-inc.com
>>> >wrote:
>>> >
>>> >> Thanks Robert. Created OOZIE-1341
>>> >>
>>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>>> >>
>>> >> >I think I follow what you're saying; assuming everything works
>>> correctly,
>>> >> >I
>>> >> >think that sounds good.  Can you make a JIRA with that and put up a
>>> >> patch;
>>> >> >that might be easier to see how this would work.
>>> >> >
>>> >> >thanks
>>> >> >- Robert
>>> >> >
>>> >> >
>>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <virag@yahoo-inc.com
>>> >
>>> >> >wrote:
>>> >> >
>>> >> >> We can keep the Main classes in the share lib, but have the action
>>> >> >> executors package the main classes for shipping it to
>>> >> >> Launcher (basically again having the method getLauncherClasses() in
>>> >> >>action
>>> >> >> executors). Also the oozie-sharelib.jar containing main classes
>>> >> >> should be part of web-app.
>>> >> >> This will improve the building/testing env as classes are in
>>> isolated
>>> >> >> sharelibs but will make the share lib optional as main classes are
>>> part
>>> >> >>of
>>> >> >> oozie-server.
>>> >> >> Can we do this change to trunk as this is blocking our QE's to pick
>>> >> >>builds
>>> >> >> due to share lib incompatibility?
>>> >> >>
>>> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g with
>>> >> Hcat
>>> >> >> integration,
>>> >> >> there can be multiple overridable implementation of URIHandlers
>>> that
>>> >> can
>>> >> >> be shipped to launcher.
>>> >> >> We can do the same for other main classes if required.
>>> >> >>
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Virag
>>> >> >>
>>> >> >>
>>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>>> >> >>
>>> >> >> >That's true, its primarily a building/testing improvement.  And
>>> from a
>>> >> >> >design perspective, it could be considered "cleaner" to not have
>>> the
>>> >> >> >launcher jar.
>>> >> >> >
>>> >> >> >Your idea sounds like a good compromise between backwards
>>> >> compatibility
>>> >> >> >and
>>> >> >> >improving the building/testing env.  How would we do that?
>>> >> >> >
>>> >> >> >As a side note, there's also a new feature with the Main classes
>>> being
>>> >> >>in
>>> >> >> >the sharelib; users can now override the Main class that Oozie
>>> will
>>> >> use
>>> >> >> >for
>>> >> >> >the action because its no longer in the launcher jar.  I think
>>> this
>>> >> >>could
>>> >> >> >still be a useful feature, but I'm guessing that if we do the
>>> >> >>compromise,
>>> >> >> >this would go away?  In any case, OOZIE-1318 is to make the Main
>>> >> >> >overridable with a config, which is probably cleaner and more
>>> flexible
>>> >> >> >than
>>> >> >> >replacing a jar file.
>>> >> >> >
>>> >> >> >thanks
>>> >> >> >- Robert
>>> >> >> >
>>> >> >> >
>>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <
>>> virag@yahoo-inc.com>
>>> >> >> >wrote:
>>> >> >> >
>>> >> >> >> That is a good suggestion for preserving compatibility, Robert.
>>> >> >> >>
>>> >> >> >> Another question is what is the main purpose of OOZIE-1311? From
>>> >> JIRA
>>> >> >> >> description, the primary
>>> >> >> >> advantage of moving Main classes from core to share lib is it
>>> can
>>> >> >>help
>>> >> >> >>in
>>> >> >> >> preventing dependencies conflicts
>>> >> >> >> (different versions of antlr's for pig, hive). However, that
>>> problem
>>> >> >> >>only
>>> >> >> >> exists while building/testing Oozie.
>>> >> >> >> If that is the case, main classes can be moved to share lib but
>>> they
>>> >> >>can
>>> >> >> >> be bundled in oozie-server itself.
>>> >> >> >> In that case, the deployment doesn't change and share lib can
>>> still
>>> >> >>be
>>> >> >> >> optional
>>> >> >> >> as oozie server would ship those classes (same as what was
>>> happening
>>> >> >> >> before). And,
>>> >> >> >> advantage of 1311 for building/testing will be retained
>>> >> >> >>
>>> >> >> >> The main concern is having share lib as a required installation
>>> will
>>> >> >> >> complicate deployment.
>>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't
>>> >> >>provide
>>> >> >> >> inbuilt support for that.
>>> >> >> >> Even though end-users wouldn't notice the change, it will be a
>>> big
>>> >> >> >>change
>>> >> >> >> for system admins installing oozie
>>> >> >> >> So even though share lib is nice feature to have, keeping it
>>> >> optional
>>> >> >> >> seems better to me as there are other ways
>>> >> >> >> Of including jars and which were introduced much before share
>>> lib.
>>> >> >> >>
>>> >> >> >> Thanks,
>>> >> >> >> Virag
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com>
>>> wrote:
>>> >> >> >>
>>> >> >> >> >Alejandro had another suggestion to prevent breaking stuff for
>>> >> >>existing
>>> >> >> >> >apps where the sharelib isn't used.  Assuming the sharelib is
>>> >> >> >>installed,
>>> >> >> >> >if
>>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then
>>> Oozie
>>> >> >>would
>>> >> >> >> >only include the oozie-*.jar from the sharelib (the one with
>>> the
>>> >> >>Main)
>>> >> >> >>in
>>> >> >> >> >it; if its set to true, then Oozie would include all of the
>>> jars in
>>> >> >>the
>>> >> >> >> >sharelib.  This would allow users to still use their own lib
>>> dir
>>> >> >>while
>>> >> >> >> >still getting the Main into the classpath.
>>> >> >> >> >
>>> >> >> >> >- Robert
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
>>> >> >> >> ><tu...@gmail.com>wrote:
>>> >> >> >> >
>>> >> >> >> >> virag, if your oozie servers dont have sharelibs currently
>>> >> >>installed
>>> >> >> >> >>then
>>> >> >> >> >> unstalling sharelibs as part of an update would not break ant
>>> >> >>running
>>> >> >> >> >>app.
>>> >> >> >> >> regarding the requirement of enabling the sharelibs to have
>>> the
>>> >> >> >>mains,
>>> >> >> >> >>you
>>> >> >> >> >> have a point. we need to figure out how to get those in the
>>> >> >> >>classpath if
>>> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave
>>> this
>>> >> >> >> >> patch in trunk for now and we move it to a release branch
>>> once we
>>> >> >> >>sorted
>>> >> >> >> >> out a way of solving this incompatibility.
>>> >> >> >> >>
>>> >> >> >> >> thanks
>>> >> >> >> >>
>>> >> >> >> >> Alejandro
>>> >> >> >> >> (phone typing)
>>> >> >> >> >>
>>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <
>>> virag@yahoo-inc.com>
>>> >> >> >>wrote:
>>> >> >> >> >>
>>> >> >> >> >> >
>>> >> >> >> >> > Tucu, Thanks for the reply and explanation.
>>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible
>>> >> >>option
>>> >> >> >>as
>>> >> >> >> >>we
>>> >> >> >> >> > cannot expect all the running actions to complete before
>>> Oozie
>>> >> >> >> >>restart.
>>> >> >> >> >> > Hot upgrade is a good option, but we will need time to
>>> >> >>implement it
>>> >> >> >> >>as we
>>> >> >> >> >> > currently don't have sharelib as part of our deployment.
>>> >> >> >> >> > So can we deprecate the refactoring of launcher classes in
>>> this
>>> >> >> >> >>release?
>>> >> >> >> >> > Also, all the launcher classes are only added to
>>> distributed
>>> >> >>cache
>>> >> >> >>if
>>> >> >> >> >>the
>>> >> >> >> >> > workflow is configured to use system lib path
>>> >> >> >> >> > (oozie.use.system.libpath=true)
>>> >> >> >> >> > We shouldn't mandate the workflow to have this property as
>>> they
>>> >> >>are
>>> >> >> >> >> > oozie's launcher classes.
>>> >> >> >> >> >
>>> >> >> >> >> > Thanks,
>>> >> >> >> >> > Virag
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tucu00@gmail.com
>>> >
>>> >> >>wrote:
>>> >> >> >> >> >
>>> >> >> >> >> >> Virag,
>>> >> >> >> >> >>
>>> >> >> >> >> >> On sharelib being required, yes you are correct. For this
>>> we
>>> >> >> >>should:
>>> >> >> >> >> >>
>>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the
>>> >> >> >>sharelib
>>> >> >> >> >>is
>>> >> >> >> >> >> REQUIRED.
>>> >> >> >> >> >>
>>> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir
>>> >> >>exists,
>>> >> >> >> >>else
>>> >> >> >> >> >> fail
>>> >> >> >> >> >> to start.
>>> >> >> >> >> >>
>>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs.
>>> >> >> >>Depending on
>>> >> >> >> >> the
>>> >> >> >> >> >> type of upgrade this may be an issue.
>>> >> >> >> >> >>
>>> >> >> >> >> >> * If you are upgrading an oozie server fix that does not
>>> >> change
>>> >> >> >>the
>>> >> >> >> >> >> sharelib files, this is not an issue and you can just
>>> shutdown
>>> >> >>the
>>> >> >> >> >>oozie
>>> >> >> >> >> >> server with in-flight jobs.
>>> >> >> >> >> >>
>>> >> >> >> >> >> * If you are upgrading an oozie server fix that involves
>>> >> >>sharelib
>>> >> >> >> >>files
>>> >> >> >> >> >> changes, then you have 2 options:
>>> >> >> >> >> >>
>>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion,
>>> wait
>>> >> >> >>till
>>> >> >> >> >>all
>>> >> >> >> >> >> running actions end, then shutdown oozie server, upgrade
>>> oozie
>>> >> >> >>server
>>> >> >> >> >> and
>>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In
>>> >> this
>>> >> >> >>case
>>> >> >> >> >>all
>>> >> >> >> >> >> new WF actions will use the new sharelib.
>>> >> >> >> >> >>
>>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the
>>> oozie-site.xml
>>> >> >> >>sharelib
>>> >> >> >> >> >> location to point to a new directory. upgrade the oozie
>>> >> server.
>>> >> >> >> >>install
>>> >> >> >> >> >> the
>>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS
>>> does
>>> >> not
>>> >> >> >> >>exist).
>>> >> >> >> >> >> start the oozie server. In this case all running WF
>>> actions
>>> >> >>will
>>> >> >> >> >> continue
>>> >> >> >> >> >> running with no issues as the JARs in the distributed
>>> cache
>>> >> >>have
>>> >> >> >>not
>>> >> >> >> >> been
>>> >> >> >> >> >> touch. All new WF actions will start using the new
>>> sharelib.
>>> >> >> >> >> >>
>>> >> >> >> >> >> Note that this sharelib upgrade protocol is not
>>> introduced by
>>> >> >> >> >>requiring
>>> >> >> >> >> >> sharelib, it is required if you have applications that use
>>> >> >> >>sharelib
>>> >> >> >> >> today.
>>> >> >> >> >> >>
>>> >> >> >> >> >> Does this address your concerns?
>>> >> >> >> >> >>
>>> >> >> >> >> >> Thanks.
>>> >> >> >> >> >>
>>> >> >> >> >> >>
>>> >> >> >> >> >>
>>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
>>> >> >> >><vi...@yahoo-inc.com>
>>> >> >> >> >> >> wrote:
>>> >> >> >> >> >>
>>> >> >> >> >> >>> Hi,
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to
>>> move all
>>> >> >>the
>>> >> >> >> >> >>> launcher
>>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to  their respective
>>> >> >> >>sharelibs.
>>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before
>>> >> >>shared
>>> >> >> >>lib
>>> >> >> >> >> was
>>> >> >> >> >> >>> optional as users could bundle jars with their workflow
>>> >> >> >>application.
>>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2
>>> problems:
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>  1.  The current deployments which don't use action
>>> shared
>>> >> lib
>>> >> >> >>will
>>> >> >> >> >> >>> fail.
>>> >> >> >> >> >>> So, probably we should deprecate the current behavior.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a
>>> job if
>>> >> >>the
>>> >> >> >> >>files
>>> >> >> >> >> >>> in
>>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running.
>>> So,
>>> >> >>when
>>> >> >> >> >>Oozie
>>> >> >> >> >> >>> is
>>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of
>>> >> >> >> >> >>>             deployment, hadoop  will fail the existing
>>> jobs
>>> >> >>for
>>> >> >> >> >>which
>>> >> >> >> >> >>> the
>>> >> >> >> >> >>> timestamp of  the file on hdfs doesn't match the
>>> timestamp of
>>> >> >>its
>>> >> >> >> >>copy
>>> >> >> >> >> >>> in
>>> >> >> >> >> >>> the job's DC.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Thanks,
>>> >> >> >> >> >>> Virag
>>> >> >> >> >> >
>>> >> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>> >
>>>
>>
>>
>
>
> --
> Alejandro
>

Re: OOZIE-1311

[Alejandro Abdelnur <tu...@cloudera.com>]

Rohini,

I understand you don't want to destabilize Oozie 4. Still, as we have no
done a 4 release yet, I think it is a good time to introduce this feature.

How about the following (similar to what we did for command interruptions)?

We do as Robert proposed in his last email, but keeping the default value
on OFF for Oozie 4; and in Oozie 5 we invert the default value to ON.

This would mean you don't have to change anything in your deployment
protocol for Oozie 4.

Thx



On Mon, May 6, 2013 at 2:30 PM, Rohini Palaniswamy
<ro...@gmail.com>wrote:

> Robert,
> > A new major release, like Oozie 4, would be a good time to introduce
> this change as the default;
>
>    We already have Oozie 4 deployed in some clusters and it is stable now.
> We are waiting for users to validate Oozie-HCat integration before we
> deploy them in production clusters and call out a release in Apache. So it
> would be good to introduce this in the next release which will also be a
> big release anyway.
>
> Regards,
> Rohini
>
>
> On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rk...@cloudera.com>wrote:
>
>> Sorry, I realize that I wasn't clear on that point.  The Main classes
>> would
>> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) but
>> the other few classes that all actions use in the launcher jar (e.g.
>> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which
>> IIRC is already included when you use sharelibs (for example, when you use
>> the sharelib with the Pig action, it includes both the share/lib/pig and
>> share/lib/oozie).
>>
>> thanks
>> - Robert
>>
>>
>> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com>
>> wrote:
>>
>> >  Hi Robert,
>> >
>> >  I didn't understand completely. Are you suggesting to have all action
>> > specific main classes in /share/lib/oozie instead of their respective
>> > action sharelibs?
>> >
>> >  Thanks,
>> > Virag
>> >
>> >   From: Robert Kanter <rk...@cloudera.com>
>> > Date: Monday, May 6, 2013 12:47 PM
>> > To: "dev@oozie.apache.org" <de...@oozie.apache.org>, Virag Kothari <
>> > virag@yahoo-inc.com>, Alejandro Abdelnur <tu...@cloudera.com>
>> > Subject: Re: OOZIE-1311
>> >
>> >   Virag,
>> > I talked more with Alejandro about the sharelib and the launcher jar.
>> >
>> > It would be good if we could eventually get rid of the launcher jar
>> > completely because:
>> >  - less files would be copied to HDFS so it should be faster
>> > - it would fix an issue where some linux distress cleanup the temp dir
>> and
>> > delete the launcher jar
>> >
>> >  The contents of the launcher jar could be moved to the Oozie sharelib
>> > (/share/lib/oozie) which IIRC currently only has a json jar file.
>> >
>> >  As for backwards compatibility, we could do something like what you
>> > suggested for the other sharelibs where the webapp module includes them.
>> >  Then we'd add some property to oozie-site that would switch between the
>> > old behavior with the launcher jar (sharelib is not required) and the
>> new
>> > behavior without the launcher jar (sharelib is required).  A new major
>> > release, like Oozie 4, would be a good time to introduce this change as
>> the
>> > default; users can use the property to go back to the old behavior.
>>  Also,
>> > if set to use the new behavior, we should make oozie.use.system.libpath
>> in
>> > the job.properties default to true because the sharelib would be
>> required,
>> > and if set to the old behavior, it would default to false.
>> >
>> >  Thoughts?
>> >
>> >  thanks
>> > - Robert
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <virag@yahoo-inc.com
>> >wrote:
>> >
>> >> Thanks Robert. Created OOZIE-1341
>> >>
>> >> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>> >>
>> >> >I think I follow what you're saying; assuming everything works
>> correctly,
>> >> >I
>> >> >think that sounds good.  Can you make a JIRA with that and put up a
>> >> patch;
>> >> >that might be easier to see how this would work.
>> >> >
>> >> >thanks
>> >> >- Robert
>> >> >
>> >> >
>> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com>
>> >> >wrote:
>> >> >
>> >> >> We can keep the Main classes in the share lib, but have the action
>> >> >> executors package the main classes for shipping it to
>> >> >> Launcher (basically again having the method getLauncherClasses() in
>> >> >>action
>> >> >> executors). Also the oozie-sharelib.jar containing main classes
>> >> >> should be part of web-app.
>> >> >> This will improve the building/testing env as classes are in
>> isolated
>> >> >> sharelibs but will make the share lib optional as main classes are
>> part
>> >> >>of
>> >> >> oozie-server.
>> >> >> Can we do this change to trunk as this is blocking our QE's to pick
>> >> >>builds
>> >> >> due to share lib incompatibility?
>> >> >>
>> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g with
>> >> Hcat
>> >> >> integration,
>> >> >> there can be multiple overridable implementation of URIHandlers that
>> >> can
>> >> >> be shipped to launcher.
>> >> >> We can do the same for other main classes if required.
>> >> >>
>> >> >>
>> >> >> Thanks,
>> >> >> Virag
>> >> >>
>> >> >>
>> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>> >> >>
>> >> >> >That's true, its primarily a building/testing improvement.  And
>> from a
>> >> >> >design perspective, it could be considered "cleaner" to not have
>> the
>> >> >> >launcher jar.
>> >> >> >
>> >> >> >Your idea sounds like a good compromise between backwards
>> >> compatibility
>> >> >> >and
>> >> >> >improving the building/testing env.  How would we do that?
>> >> >> >
>> >> >> >As a side note, there's also a new feature with the Main classes
>> being
>> >> >>in
>> >> >> >the sharelib; users can now override the Main class that Oozie will
>> >> use
>> >> >> >for
>> >> >> >the action because its no longer in the launcher jar.  I think this
>> >> >>could
>> >> >> >still be a useful feature, but I'm guessing that if we do the
>> >> >>compromise,
>> >> >> >this would go away?  In any case, OOZIE-1318 is to make the Main
>> >> >> >overridable with a config, which is probably cleaner and more
>> flexible
>> >> >> >than
>> >> >> >replacing a jar file.
>> >> >> >
>> >> >> >thanks
>> >> >> >- Robert
>> >> >> >
>> >> >> >
>> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <
>> virag@yahoo-inc.com>
>> >> >> >wrote:
>> >> >> >
>> >> >> >> That is a good suggestion for preserving compatibility, Robert.
>> >> >> >>
>> >> >> >> Another question is what is the main purpose of OOZIE-1311? From
>> >> JIRA
>> >> >> >> description, the primary
>> >> >> >> advantage of moving Main classes from core to share lib is it can
>> >> >>help
>> >> >> >>in
>> >> >> >> preventing dependencies conflicts
>> >> >> >> (different versions of antlr's for pig, hive). However, that
>> problem
>> >> >> >>only
>> >> >> >> exists while building/testing Oozie.
>> >> >> >> If that is the case, main classes can be moved to share lib but
>> they
>> >> >>can
>> >> >> >> be bundled in oozie-server itself.
>> >> >> >> In that case, the deployment doesn't change and share lib can
>> still
>> >> >>be
>> >> >> >> optional
>> >> >> >> as oozie server would ship those classes (same as what was
>> happening
>> >> >> >> before). And,
>> >> >> >> advantage of 1311 for building/testing will be retained
>> >> >> >>
>> >> >> >> The main concern is having share lib as a required installation
>> will
>> >> >> >> complicate deployment.
>> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't
>> >> >>provide
>> >> >> >> inbuilt support for that.
>> >> >> >> Even though end-users wouldn't notice the change, it will be a
>> big
>> >> >> >>change
>> >> >> >> for system admins installing oozie
>> >> >> >> So even though share lib is nice feature to have, keeping it
>> >> optional
>> >> >> >> seems better to me as there are other ways
>> >> >> >> Of including jars and which were introduced much before share
>> lib.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> Virag
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com>
>> wrote:
>> >> >> >>
>> >> >> >> >Alejandro had another suggestion to prevent breaking stuff for
>> >> >>existing
>> >> >> >> >apps where the sharelib isn't used.  Assuming the sharelib is
>> >> >> >>installed,
>> >> >> >> >if
>> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then
>> Oozie
>> >> >>would
>> >> >> >> >only include the oozie-*.jar from the sharelib (the one with the
>> >> >>Main)
>> >> >> >>in
>> >> >> >> >it; if its set to true, then Oozie would include all of the
>> jars in
>> >> >>the
>> >> >> >> >sharelib.  This would allow users to still use their own lib dir
>> >> >>while
>> >> >> >> >still getting the Main into the classpath.
>> >> >> >> >
>> >> >> >> >- Robert
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
>> >> >> >> ><tu...@gmail.com>wrote:
>> >> >> >> >
>> >> >> >> >> virag, if your oozie servers dont have sharelibs currently
>> >> >>installed
>> >> >> >> >>then
>> >> >> >> >> unstalling sharelibs as part of an update would not break ant
>> >> >>running
>> >> >> >> >>app.
>> >> >> >> >> regarding the requirement of enabling the sharelibs to have
>> the
>> >> >> >>mains,
>> >> >> >> >>you
>> >> >> >> >> have a point. we need to figure out how to get those in the
>> >> >> >>classpath if
>> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave this
>> >> >> >> >> patch in trunk for now and we move it to a release branch
>> once we
>> >> >> >>sorted
>> >> >> >> >> out a way of solving this incompatibility.
>> >> >> >> >>
>> >> >> >> >> thanks
>> >> >> >> >>
>> >> >> >> >> Alejandro
>> >> >> >> >> (phone typing)
>> >> >> >> >>
>> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <
>> virag@yahoo-inc.com>
>> >> >> >>wrote:
>> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > Tucu, Thanks for the reply and explanation.
>> >> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible
>> >> >>option
>> >> >> >>as
>> >> >> >> >>we
>> >> >> >> >> > cannot expect all the running actions to complete before
>> Oozie
>> >> >> >> >>restart.
>> >> >> >> >> > Hot upgrade is a good option, but we will need time to
>> >> >>implement it
>> >> >> >> >>as we
>> >> >> >> >> > currently don't have sharelib as part of our deployment.
>> >> >> >> >> > So can we deprecate the refactoring of launcher classes in
>> this
>> >> >> >> >>release?
>> >> >> >> >> > Also, all the launcher classes are only added to distributed
>> >> >>cache
>> >> >> >>if
>> >> >> >> >>the
>> >> >> >> >> > workflow is configured to use system lib path
>> >> >> >> >> > (oozie.use.system.libpath=true)
>> >> >> >> >> > We shouldn't mandate the workflow to have this property as
>> they
>> >> >>are
>> >> >> >> >> > oozie's launcher classes.
>> >> >> >> >> >
>> >> >> >> >> > Thanks,
>> >> >> >> >> > Virag
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tu...@gmail.com>
>> >> >>wrote:
>> >> >> >> >> >
>> >> >> >> >> >> Virag,
>> >> >> >> >> >>
>> >> >> >> >> >> On sharelib being required, yes you are correct. For this
>> we
>> >> >> >>should:
>> >> >> >> >> >>
>> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the
>> >> >> >>sharelib
>> >> >> >> >>is
>> >> >> >> >> >> REQUIRED.
>> >> >> >> >> >>
>> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir
>> >> >>exists,
>> >> >> >> >>else
>> >> >> >> >> >> fail
>> >> >> >> >> >> to start.
>> >> >> >> >> >>
>> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs.
>> >> >> >>Depending on
>> >> >> >> >> the
>> >> >> >> >> >> type of upgrade this may be an issue.
>> >> >> >> >> >>
>> >> >> >> >> >> * If you are upgrading an oozie server fix that does not
>> >> change
>> >> >> >>the
>> >> >> >> >> >> sharelib files, this is not an issue and you can just
>> shutdown
>> >> >>the
>> >> >> >> >>oozie
>> >> >> >> >> >> server with in-flight jobs.
>> >> >> >> >> >>
>> >> >> >> >> >> * If you are upgrading an oozie server fix that involves
>> >> >>sharelib
>> >> >> >> >>files
>> >> >> >> >> >> changes, then you have 2 options:
>> >> >> >> >> >>
>> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion,
>> wait
>> >> >> >>till
>> >> >> >> >>all
>> >> >> >> >> >> running actions end, then shutdown oozie server, upgrade
>> oozie
>> >> >> >>server
>> >> >> >> >> and
>> >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In
>> >> this
>> >> >> >>case
>> >> >> >> >>all
>> >> >> >> >> >> new WF actions will use the new sharelib.
>> >> >> >> >> >>
>> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the
>> oozie-site.xml
>> >> >> >>sharelib
>> >> >> >> >> >> location to point to a new directory. upgrade the oozie
>> >> server.
>> >> >> >> >>install
>> >> >> >> >> >> the
>> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does
>> >> not
>> >> >> >> >>exist).
>> >> >> >> >> >> start the oozie server. In this case all running WF actions
>> >> >>will
>> >> >> >> >> continue
>> >> >> >> >> >> running with no issues as the JARs in the distributed cache
>> >> >>have
>> >> >> >>not
>> >> >> >> >> been
>> >> >> >> >> >> touch. All new WF actions will start using the new
>> sharelib.
>> >> >> >> >> >>
>> >> >> >> >> >> Note that this sharelib upgrade protocol is not introduced
>> by
>> >> >> >> >>requiring
>> >> >> >> >> >> sharelib, it is required if you have applications that use
>> >> >> >>sharelib
>> >> >> >> >> today.
>> >> >> >> >> >>
>> >> >> >> >> >> Does this address your concerns?
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks.
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
>> >> >> >><vi...@yahoo-inc.com>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>
>> >> >> >> >> >>> Hi,
>> >> >> >> >> >>>
>> >> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to move
>> all
>> >> >>the
>> >> >> >> >> >>> launcher
>> >> >> >> >> >>> classes like PigMain, HiveMain etc. to  their respective
>> >> >> >>sharelibs.
>> >> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before
>> >> >>shared
>> >> >> >>lib
>> >> >> >> >> was
>> >> >> >> >> >>> optional as users could bundle jars with their workflow
>> >> >> >>application.
>> >> >> >> >> >>> So always requiring shared lib seems to introduce 2
>> problems:
>> >> >> >> >> >>>
>> >> >> >> >> >>>  1.  The current deployments which don't use action shared
>> >> lib
>> >> >> >>will
>> >> >> >> >> >>> fail.
>> >> >> >> >> >>> So, probably we should deprecate the current behavior.
>> >> >> >> >> >>>
>> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job
>> if
>> >> >>the
>> >> >> >> >>files
>> >> >> >> >> >>> in
>> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running.
>> So,
>> >> >>when
>> >> >> >> >>Oozie
>> >> >> >> >> >>> is
>> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of
>> >> >> >> >> >>>             deployment, hadoop  will fail the existing
>> jobs
>> >> >>for
>> >> >> >> >>which
>> >> >> >> >> >>> the
>> >> >> >> >> >>> timestamp of  the file on hdfs doesn't match the
>> timestamp of
>> >> >>its
>> >> >> >> >>copy
>> >> >> >> >> >>> in
>> >> >> >> >> >>> the job's DC.
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> Thanks,
>> >> >> >> >> >>> Virag
>> >> >> >> >> >
>> >> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>> >
>>
>
>


-- 
Alejandro

Re: OOZIE-1311

[Rohini Palaniswamy <ro...@gmail.com>]

Robert,
> A new major release, like Oozie 4, would be a good time to introduce this
change as the default;

   We already have Oozie 4 deployed in some clusters and it is stable now.
We are waiting for users to validate Oozie-HCat integration before we
deploy them in production clusters and call out a release in Apache. So it
would be good to introduce this in the next release which will also be a
big release anyway.

Regards,
Rohini


On Mon, May 6, 2013 at 2:09 PM, Robert Kanter <rk...@cloudera.com> wrote:

> Sorry, I realize that I wasn't clear on that point.  The Main classes would
> go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) but
> the other few classes that all actions use in the launcher jar (e.g.
> LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which
> IIRC is already included when you use sharelibs (for example, when you use
> the sharelib with the Pig action, it includes both the share/lib/pig and
> share/lib/oozie).
>
> thanks
> - Robert
>
>
> On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com> wrote:
>
> >  Hi Robert,
> >
> >  I didn't understand completely. Are you suggesting to have all action
> > specific main classes in /share/lib/oozie instead of their respective
> > action sharelibs?
> >
> >  Thanks,
> > Virag
> >
> >   From: Robert Kanter <rk...@cloudera.com>
> > Date: Monday, May 6, 2013 12:47 PM
> > To: "dev@oozie.apache.org" <de...@oozie.apache.org>, Virag Kothari <
> > virag@yahoo-inc.com>, Alejandro Abdelnur <tu...@cloudera.com>
> > Subject: Re: OOZIE-1311
> >
> >   Virag,
> > I talked more with Alejandro about the sharelib and the launcher jar.
> >
> > It would be good if we could eventually get rid of the launcher jar
> > completely because:
> >  - less files would be copied to HDFS so it should be faster
> > - it would fix an issue where some linux distress cleanup the temp dir
> and
> > delete the launcher jar
> >
> >  The contents of the launcher jar could be moved to the Oozie sharelib
> > (/share/lib/oozie) which IIRC currently only has a json jar file.
> >
> >  As for backwards compatibility, we could do something like what you
> > suggested for the other sharelibs where the webapp module includes them.
> >  Then we'd add some property to oozie-site that would switch between the
> > old behavior with the launcher jar (sharelib is not required) and the new
> > behavior without the launcher jar (sharelib is required).  A new major
> > release, like Oozie 4, would be a good time to introduce this change as
> the
> > default; users can use the property to go back to the old behavior.
>  Also,
> > if set to use the new behavior, we should make oozie.use.system.libpath
> in
> > the job.properties default to true because the sharelib would be
> required,
> > and if set to the old behavior, it would default to false.
> >
> >  Thoughts?
> >
> >  thanks
> > - Robert
> >
> >
> >
> >
> >
> > On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <virag@yahoo-inc.com
> >wrote:
> >
> >> Thanks Robert. Created OOZIE-1341
> >>
> >> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
> >>
> >> >I think I follow what you're saying; assuming everything works
> correctly,
> >> >I
> >> >think that sounds good.  Can you make a JIRA with that and put up a
> >> patch;
> >> >that might be easier to see how this would work.
> >> >
> >> >thanks
> >> >- Robert
> >> >
> >> >
> >> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com>
> >> >wrote:
> >> >
> >> >> We can keep the Main classes in the share lib, but have the action
> >> >> executors package the main classes for shipping it to
> >> >> Launcher (basically again having the method getLauncherClasses() in
> >> >>action
> >> >> executors). Also the oozie-sharelib.jar containing main classes
> >> >> should be part of web-app.
> >> >> This will improve the building/testing env as classes are in isolated
> >> >> sharelibs but will make the share lib optional as main classes are
> part
> >> >>of
> >> >> oozie-server.
> >> >> Can we do this change to trunk as this is blocking our QE's to pick
> >> >>builds
> >> >> due to share lib incompatibility?
> >> >>
> >> >> We can still do what oozie-1318 is trying to achieve. For, e.g with
> >> Hcat
> >> >> integration,
> >> >> there can be multiple overridable implementation of URIHandlers that
> >> can
> >> >> be shipped to launcher.
> >> >> We can do the same for other main classes if required.
> >> >>
> >> >>
> >> >> Thanks,
> >> >> Virag
> >> >>
> >> >>
> >> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
> >> >>
> >> >> >That's true, its primarily a building/testing improvement.  And
> from a
> >> >> >design perspective, it could be considered "cleaner" to not have the
> >> >> >launcher jar.
> >> >> >
> >> >> >Your idea sounds like a good compromise between backwards
> >> compatibility
> >> >> >and
> >> >> >improving the building/testing env.  How would we do that?
> >> >> >
> >> >> >As a side note, there's also a new feature with the Main classes
> being
> >> >>in
> >> >> >the sharelib; users can now override the Main class that Oozie will
> >> use
> >> >> >for
> >> >> >the action because its no longer in the launcher jar.  I think this
> >> >>could
> >> >> >still be a useful feature, but I'm guessing that if we do the
> >> >>compromise,
> >> >> >this would go away?  In any case, OOZIE-1318 is to make the Main
> >> >> >overridable with a config, which is probably cleaner and more
> flexible
> >> >> >than
> >> >> >replacing a jar file.
> >> >> >
> >> >> >thanks
> >> >> >- Robert
> >> >> >
> >> >> >
> >> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <virag@yahoo-inc.com
> >
> >> >> >wrote:
> >> >> >
> >> >> >> That is a good suggestion for preserving compatibility, Robert.
> >> >> >>
> >> >> >> Another question is what is the main purpose of OOZIE-1311? From
> >> JIRA
> >> >> >> description, the primary
> >> >> >> advantage of moving Main classes from core to share lib is it can
> >> >>help
> >> >> >>in
> >> >> >> preventing dependencies conflicts
> >> >> >> (different versions of antlr's for pig, hive). However, that
> problem
> >> >> >>only
> >> >> >> exists while building/testing Oozie.
> >> >> >> If that is the case, main classes can be moved to share lib but
> they
> >> >>can
> >> >> >> be bundled in oozie-server itself.
> >> >> >> In that case, the deployment doesn't change and share lib can
> still
> >> >>be
> >> >> >> optional
> >> >> >> as oozie server would ship those classes (same as what was
> happening
> >> >> >> before). And,
> >> >> >> advantage of 1311 for building/testing will be retained
> >> >> >>
> >> >> >> The main concern is having share lib as a required installation
> will
> >> >> >> complicate deployment.
> >> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't
> >> >>provide
> >> >> >> inbuilt support for that.
> >> >> >> Even though end-users wouldn't notice the change, it will be a big
> >> >> >>change
> >> >> >> for system admins installing oozie
> >> >> >> So even though share lib is nice feature to have, keeping it
> >> optional
> >> >> >> seems better to me as there are other ways
> >> >> >> Of including jars and which were introduced much before share lib.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> Virag
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
> >> >> >>
> >> >> >> >Alejandro had another suggestion to prevent breaking stuff for
> >> >>existing
> >> >> >> >apps where the sharelib isn't used.  Assuming the sharelib is
> >> >> >>installed,
> >> >> >> >if
> >> >> >> >the user doesn't set oozie.use.system.libpath to true, then Oozie
> >> >>would
> >> >> >> >only include the oozie-*.jar from the sharelib (the one with the
> >> >>Main)
> >> >> >>in
> >> >> >> >it; if its set to true, then Oozie would include all of the jars
> in
> >> >>the
> >> >> >> >sharelib.  This would allow users to still use their own lib dir
> >> >>while
> >> >> >> >still getting the Main into the classpath.
> >> >> >> >
> >> >> >> >- Robert
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
> >> >> >> ><tu...@gmail.com>wrote:
> >> >> >> >
> >> >> >> >> virag, if your oozie servers dont have sharelibs currently
> >> >>installed
> >> >> >> >>then
> >> >> >> >> unstalling sharelibs as part of an update would not break ant
> >> >>running
> >> >> >> >>app.
> >> >> >> >> regarding the requirement of enabling the sharelibs to have the
> >> >> >>mains,
> >> >> >> >>you
> >> >> >> >> have a point. we need to figure out how to get those in the
> >> >> >>classpath if
> >> >> >> >> the user is not using the sharelibs. i'd suggest we leave this
> >> >> >> >> patch in trunk for now and we move it to a release branch once
> we
> >> >> >>sorted
> >> >> >> >> out a way of solving this incompatibility.
> >> >> >> >>
> >> >> >> >> thanks
> >> >> >> >>
> >> >> >> >> Alejandro
> >> >> >> >> (phone typing)
> >> >> >> >>
> >> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <
> virag@yahoo-inc.com>
> >> >> >>wrote:
> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> > Tucu, Thanks for the reply and explanation.
> >> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible
> >> >>option
> >> >> >>as
> >> >> >> >>we
> >> >> >> >> > cannot expect all the running actions to complete before
> Oozie
> >> >> >> >>restart.
> >> >> >> >> > Hot upgrade is a good option, but we will need time to
> >> >>implement it
> >> >> >> >>as we
> >> >> >> >> > currently don't have sharelib as part of our deployment.
> >> >> >> >> > So can we deprecate the refactoring of launcher classes in
> this
> >> >> >> >>release?
> >> >> >> >> > Also, all the launcher classes are only added to distributed
> >> >>cache
> >> >> >>if
> >> >> >> >>the
> >> >> >> >> > workflow is configured to use system lib path
> >> >> >> >> > (oozie.use.system.libpath=true)
> >> >> >> >> > We shouldn't mandate the workflow to have this property as
> they
> >> >>are
> >> >> >> >> > oozie's launcher classes.
> >> >> >> >> >
> >> >> >> >> > Thanks,
> >> >> >> >> > Virag
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tu...@gmail.com>
> >> >>wrote:
> >> >> >> >> >
> >> >> >> >> >> Virag,
> >> >> >> >> >>
> >> >> >> >> >> On sharelib being required, yes you are correct. For this we
> >> >> >>should:
> >> >> >> >> >>
> >> >> >> >> >> * Make 100% clear in the quick-start/install docs that the
> >> >> >>sharelib
> >> >> >> >>is
> >> >> >> >> >> REQUIRED.
> >> >> >> >> >>
> >> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir
> >> >>exists,
> >> >> >> >>else
> >> >> >> >> >> fail
> >> >> >> >> >> to start.
> >> >> >> >> >>
> >> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs.
> >> >> >>Depending on
> >> >> >> >> the
> >> >> >> >> >> type of upgrade this may be an issue.
> >> >> >> >> >>
> >> >> >> >> >> * If you are upgrading an oozie server fix that does not
> >> change
> >> >> >>the
> >> >> >> >> >> sharelib files, this is not an issue and you can just
> shutdown
> >> >>the
> >> >> >> >>oozie
> >> >> >> >> >> server with in-flight jobs.
> >> >> >> >> >>
> >> >> >> >> >> * If you are upgrading an oozie server fix that involves
> >> >>sharelib
> >> >> >> >>files
> >> >> >> >> >> changes, then you have 2 options:
> >> >> >> >> >>
> >> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion,
> wait
> >> >> >>till
> >> >> >> >>all
> >> >> >> >> >> running actions end, then shutdown oozie server, upgrade
> oozie
> >> >> >>server
> >> >> >> >> and
> >> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In
> >> this
> >> >> >>case
> >> >> >> >>all
> >> >> >> >> >> new WF actions will use the new sharelib.
> >> >> >> >> >>
> >> >> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml
> >> >> >>sharelib
> >> >> >> >> >> location to point to a new directory. upgrade the oozie
> >> server.
> >> >> >> >>install
> >> >> >> >> >> the
> >> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does
> >> not
> >> >> >> >>exist).
> >> >> >> >> >> start the oozie server. In this case all running WF actions
> >> >>will
> >> >> >> >> continue
> >> >> >> >> >> running with no issues as the JARs in the distributed cache
> >> >>have
> >> >> >>not
> >> >> >> >> been
> >> >> >> >> >> touch. All new WF actions will start using the new sharelib.
> >> >> >> >> >>
> >> >> >> >> >> Note that this sharelib upgrade protocol is not introduced
> by
> >> >> >> >>requiring
> >> >> >> >> >> sharelib, it is required if you have applications that use
> >> >> >>sharelib
> >> >> >> >> today.
> >> >> >> >> >>
> >> >> >> >> >> Does this address your concerns?
> >> >> >> >> >>
> >> >> >> >> >> Thanks.
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
> >> >> >><vi...@yahoo-inc.com>
> >> >> >> >> >> wrote:
> >> >> >> >> >>
> >> >> >> >> >>> Hi,
> >> >> >> >> >>>
> >> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to move
> all
> >> >>the
> >> >> >> >> >>> launcher
> >> >> >> >> >>> classes like PigMain, HiveMain etc. to  their respective
> >> >> >>sharelibs.
> >> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before
> >> >>shared
> >> >> >>lib
> >> >> >> >> was
> >> >> >> >> >>> optional as users could bundle jars with their workflow
> >> >> >>application.
> >> >> >> >> >>> So always requiring shared lib seems to introduce 2
> problems:
> >> >> >> >> >>>
> >> >> >> >> >>>  1.  The current deployments which don't use action shared
> >> lib
> >> >> >>will
> >> >> >> >> >>> fail.
> >> >> >> >> >>> So, probably we should deprecate the current behavior.
> >> >> >> >> >>>
> >> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job
> if
> >> >>the
> >> >> >> >>files
> >> >> >> >> >>> in
> >> >> >> >> >>> DC are updated on hdfs while the hadoop job is running. So,
> >> >>when
> >> >> >> >>Oozie
> >> >> >> >> >>> is
> >> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of
> >> >> >> >> >>>             deployment, hadoop  will fail the existing jobs
> >> >>for
> >> >> >> >>which
> >> >> >> >> >>> the
> >> >> >> >> >>> timestamp of  the file on hdfs doesn't match the timestamp
> of
> >> >>its
> >> >> >> >>copy
> >> >> >> >> >>> in
> >> >> >> >> >>> the job's DC.
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> Thanks,
> >> >> >> >> >>> Virag
> >> >> >> >> >
> >> >> >> >>
> >> >> >>
> >> >> >>
> >> >>
> >> >>
> >>
> >>
> >
>

Re: OOZIE-1311

[Robert Kanter <rk...@cloudera.com>]

Sorry, I realize that I wasn't clear on that point.  The Main classes would
go in their respective sharelib (e.g. PigMain goes in share/lib/pig/) but
the other few classes that all actions use in the launcher jar (e.g.
LauncherMapper) would go in the oozie sharelib (share/lib/oozie), which
IIRC is already included when you use sharelibs (for example, when you use
the sharelib with the Pig action, it includes both the share/lib/pig and
share/lib/oozie).

thanks
- Robert


On Mon, May 6, 2013 at 1:52 PM, Virag Kothari <vi...@yahoo-inc.com> wrote:

>  Hi Robert,
>
>  I didn't understand completely. Are you suggesting to have all action
> specific main classes in /share/lib/oozie instead of their respective
> action sharelibs?
>
>  Thanks,
> Virag
>
>   From: Robert Kanter <rk...@cloudera.com>
> Date: Monday, May 6, 2013 12:47 PM
> To: "dev@oozie.apache.org" <de...@oozie.apache.org>, Virag Kothari <
> virag@yahoo-inc.com>, Alejandro Abdelnur <tu...@cloudera.com>
> Subject: Re: OOZIE-1311
>
>   Virag,
> I talked more with Alejandro about the sharelib and the launcher jar.
>
> It would be good if we could eventually get rid of the launcher jar
> completely because:
>  - less files would be copied to HDFS so it should be faster
> - it would fix an issue where some linux distress cleanup the temp dir and
> delete the launcher jar
>
>  The contents of the launcher jar could be moved to the Oozie sharelib
> (/share/lib/oozie) which IIRC currently only has a json jar file.
>
>  As for backwards compatibility, we could do something like what you
> suggested for the other sharelibs where the webapp module includes them.
>  Then we'd add some property to oozie-site that would switch between the
> old behavior with the launcher jar (sharelib is not required) and the new
> behavior without the launcher jar (sharelib is required).  A new major
> release, like Oozie 4, would be a good time to introduce this change as the
> default; users can use the property to go back to the old behavior.  Also,
> if set to use the new behavior, we should make oozie.use.system.libpath in
> the job.properties default to true because the sharelib would be required,
> and if set to the old behavior, it would default to false.
>
>  Thoughts?
>
>  thanks
> - Robert
>
>
>
>
>
> On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <vi...@yahoo-inc.com>wrote:
>
>> Thanks Robert. Created OOZIE-1341
>>
>> On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>>
>> >I think I follow what you're saying; assuming everything works correctly,
>> >I
>> >think that sounds good.  Can you make a JIRA with that and put up a
>> patch;
>> >that might be easier to see how this would work.
>> >
>> >thanks
>> >- Robert
>> >
>> >
>> >On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com>
>> >wrote:
>> >
>> >> We can keep the Main classes in the share lib, but have the action
>> >> executors package the main classes for shipping it to
>> >> Launcher (basically again having the method getLauncherClasses() in
>> >>action
>> >> executors). Also the oozie-sharelib.jar containing main classes
>> >> should be part of web-app.
>> >> This will improve the building/testing env as classes are in isolated
>> >> sharelibs but will make the share lib optional as main classes are part
>> >>of
>> >> oozie-server.
>> >> Can we do this change to trunk as this is blocking our QE's to pick
>> >>builds
>> >> due to share lib incompatibility?
>> >>
>> >> We can still do what oozie-1318 is trying to achieve. For, e.g with
>> Hcat
>> >> integration,
>> >> there can be multiple overridable implementation of URIHandlers that
>> can
>> >> be shipped to launcher.
>> >> We can do the same for other main classes if required.
>> >>
>> >>
>> >> Thanks,
>> >> Virag
>> >>
>> >>
>> >> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>> >>
>> >> >That's true, its primarily a building/testing improvement.  And from a
>> >> >design perspective, it could be considered "cleaner" to not have the
>> >> >launcher jar.
>> >> >
>> >> >Your idea sounds like a good compromise between backwards
>> compatibility
>> >> >and
>> >> >improving the building/testing env.  How would we do that?
>> >> >
>> >> >As a side note, there's also a new feature with the Main classes being
>> >>in
>> >> >the sharelib; users can now override the Main class that Oozie will
>> use
>> >> >for
>> >> >the action because its no longer in the launcher jar.  I think this
>> >>could
>> >> >still be a useful feature, but I'm guessing that if we do the
>> >>compromise,
>> >> >this would go away?  In any case, OOZIE-1318 is to make the Main
>> >> >overridable with a config, which is probably cleaner and more flexible
>> >> >than
>> >> >replacing a jar file.
>> >> >
>> >> >thanks
>> >> >- Robert
>> >> >
>> >> >
>> >> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <vi...@yahoo-inc.com>
>> >> >wrote:
>> >> >
>> >> >> That is a good suggestion for preserving compatibility, Robert.
>> >> >>
>> >> >> Another question is what is the main purpose of OOZIE-1311? From
>> JIRA
>> >> >> description, the primary
>> >> >> advantage of moving Main classes from core to share lib is it can
>> >>help
>> >> >>in
>> >> >> preventing dependencies conflicts
>> >> >> (different versions of antlr's for pig, hive). However, that problem
>> >> >>only
>> >> >> exists while building/testing Oozie.
>> >> >> If that is the case, main classes can be moved to share lib but they
>> >>can
>> >> >> be bundled in oozie-server itself.
>> >> >> In that case, the deployment doesn't change and share lib can still
>> >>be
>> >> >> optional
>> >> >> as oozie server would ship those classes (same as what was happening
>> >> >> before). And,
>> >> >> advantage of 1311 for building/testing will be retained
>> >> >>
>> >> >> The main concern is having share lib as a required installation will
>> >> >> complicate deployment.
>> >> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't
>> >>provide
>> >> >> inbuilt support for that.
>> >> >> Even though end-users wouldn't notice the change, it will be a big
>> >> >>change
>> >> >> for system admins installing oozie
>> >> >> So even though share lib is nice feature to have, keeping it
>> optional
>> >> >> seems better to me as there are other ways
>> >> >> Of including jars and which were introduced much before share lib.
>> >> >>
>> >> >> Thanks,
>> >> >> Virag
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com> wrote:
>> >> >>
>> >> >> >Alejandro had another suggestion to prevent breaking stuff for
>> >>existing
>> >> >> >apps where the sharelib isn't used.  Assuming the sharelib is
>> >> >>installed,
>> >> >> >if
>> >> >> >the user doesn't set oozie.use.system.libpath to true, then Oozie
>> >>would
>> >> >> >only include the oozie-*.jar from the sharelib (the one with the
>> >>Main)
>> >> >>in
>> >> >> >it; if its set to true, then Oozie would include all of the jars in
>> >>the
>> >> >> >sharelib.  This would allow users to still use their own lib dir
>> >>while
>> >> >> >still getting the Main into the classpath.
>> >> >> >
>> >> >> >- Robert
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
>> >> >> ><tu...@gmail.com>wrote:
>> >> >> >
>> >> >> >> virag, if your oozie servers dont have sharelibs currently
>> >>installed
>> >> >> >>then
>> >> >> >> unstalling sharelibs as part of an update would not break ant
>> >>running
>> >> >> >>app.
>> >> >> >> regarding the requirement of enabling the sharelibs to have the
>> >> >>mains,
>> >> >> >>you
>> >> >> >> have a point. we need to figure out how to get those in the
>> >> >>classpath if
>> >> >> >> the user is not using the sharelibs. i'd suggest we leave this
>> >> >> >> patch in trunk for now and we move it to a release branch once we
>> >> >>sorted
>> >> >> >> out a way of solving this incompatibility.
>> >> >> >>
>> >> >> >> thanks
>> >> >> >>
>> >> >> >> Alejandro
>> >> >> >> (phone typing)
>> >> >> >>
>> >> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <vi...@yahoo-inc.com>
>> >> >>wrote:
>> >> >> >>
>> >> >> >> >
>> >> >> >> > Tucu, Thanks for the reply and explanation.
>> >> >> >> > For workflows running in Y!, cold upgrade is not a feasible
>> >>option
>> >> >>as
>> >> >> >>we
>> >> >> >> > cannot expect all the running actions to complete before Oozie
>> >> >> >>restart.
>> >> >> >> > Hot upgrade is a good option, but we will need time to
>> >>implement it
>> >> >> >>as we
>> >> >> >> > currently don't have sharelib as part of our deployment.
>> >> >> >> > So can we deprecate the refactoring of launcher classes in this
>> >> >> >>release?
>> >> >> >> > Also, all the launcher classes are only added to distributed
>> >>cache
>> >> >>if
>> >> >> >>the
>> >> >> >> > workflow is configured to use system lib path
>> >> >> >> > (oozie.use.system.libpath=true)
>> >> >> >> > We shouldn't mandate the workflow to have this property as they
>> >>are
>> >> >> >> > oozie's launcher classes.
>> >> >> >> >
>> >> >> >> > Thanks,
>> >> >> >> > Virag
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tu...@gmail.com>
>> >>wrote:
>> >> >> >> >
>> >> >> >> >> Virag,
>> >> >> >> >>
>> >> >> >> >> On sharelib being required, yes you are correct. For this we
>> >> >>should:
>> >> >> >> >>
>> >> >> >> >> * Make 100% clear in the quick-start/install docs that the
>> >> >>sharelib
>> >> >> >>is
>> >> >> >> >> REQUIRED.
>> >> >> >> >>
>> >> >> >> >> * Add a check at Oozie startup to verify the sharelib dir
>> >>exists,
>> >> >> >>else
>> >> >> >> >> fail
>> >> >> >> >> to start.
>> >> >> >> >>
>> >> >> >> >> On sharelib lib issue during upgrade for in-flight jobs.
>> >> >>Depending on
>> >> >> >> the
>> >> >> >> >> type of upgrade this may be an issue.
>> >> >> >> >>
>> >> >> >> >> * If you are upgrading an oozie server fix that does not
>> change
>> >> >>the
>> >> >> >> >> sharelib files, this is not an issue and you can just shutdown
>> >>the
>> >> >> >>oozie
>> >> >> >> >> server with in-flight jobs.
>> >> >> >> >>
>> >> >> >> >> * If you are upgrading an oozie server fix that involves
>> >>sharelib
>> >> >> >>files
>> >> >> >> >> changes, then you have 2 options:
>> >> >> >> >>
>> >> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait
>> >> >>till
>> >> >> >>all
>> >> >> >> >> running actions end, then shutdown oozie server, upgrade oozie
>> >> >>server
>> >> >> >> and
>> >> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In
>> this
>> >> >>case
>> >> >> >>all
>> >> >> >> >> new WF actions will use the new sharelib.
>> >> >> >> >>
>> >> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml
>> >> >>sharelib
>> >> >> >> >> location to point to a new directory. upgrade the oozie
>> server.
>> >> >> >>install
>> >> >> >> >> the
>> >> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does
>> not
>> >> >> >>exist).
>> >> >> >> >> start the oozie server. In this case all running WF actions
>> >>will
>> >> >> >> continue
>> >> >> >> >> running with no issues as the JARs in the distributed cache
>> >>have
>> >> >>not
>> >> >> >> been
>> >> >> >> >> touch. All new WF actions will start using the new sharelib.
>> >> >> >> >>
>> >> >> >> >> Note that this sharelib upgrade protocol is not introduced by
>> >> >> >>requiring
>> >> >> >> >> sharelib, it is required if you have applications that use
>> >> >>sharelib
>> >> >> >> today.
>> >> >> >> >>
>> >> >> >> >> Does this address your concerns?
>> >> >> >> >>
>> >> >> >> >> Thanks.
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
>> >> >><vi...@yahoo-inc.com>
>> >> >> >> >> wrote:
>> >> >> >> >>
>> >> >> >> >>> Hi,
>> >> >> >> >>>
>> >> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to move all
>> >>the
>> >> >> >> >>> launcher
>> >> >> >> >>> classes like PigMain, HiveMain etc. to  their respective
>> >> >>sharelibs.
>> >> >> >> >>> So, now shared lib is a mandatory deployment step. Before
>> >>shared
>> >> >>lib
>> >> >> >> was
>> >> >> >> >>> optional as users could bundle jars with their workflow
>> >> >>application.
>> >> >> >> >>> So always requiring shared lib seems to introduce 2 problems:
>> >> >> >> >>>
>> >> >> >> >>>  1.  The current deployments which don't use action shared
>> lib
>> >> >>will
>> >> >> >> >>> fail.
>> >> >> >> >>> So, probably we should deprecate the current behavior.
>> >> >> >> >>>
>> >> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job if
>> >>the
>> >> >> >>files
>> >> >> >> >>> in
>> >> >> >> >>> DC are updated on hdfs while the hadoop job is running. So,
>> >>when
>> >> >> >>Oozie
>> >> >> >> >>> is
>> >> >> >> >>> restarted and shared lib is uploaded to hdfs as part of
>> >> >> >> >>>             deployment, hadoop  will fail the existing jobs
>> >>for
>> >> >> >>which
>> >> >> >> >>> the
>> >> >> >> >>> timestamp of  the file on hdfs doesn't match the timestamp of
>> >>its
>> >> >> >>copy
>> >> >> >> >>> in
>> >> >> >> >>> the job's DC.
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>> Thanks,
>> >> >> >> >>> Virag
>> >> >> >> >
>> >> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>
>

Re: OOZIE-1311

[Virag Kothari <vi...@yahoo-inc.com>]

Hi Robert,

I didn't understand completely. Are you suggesting to have all action specific main classes in /share/lib/oozie instead of their respective action sharelibs?

Thanks,
Virag

From: Robert Kanter <rk...@cloudera.com>>
Date: Monday, May 6, 2013 12:47 PM
To: "dev@oozie.apache.org<ma...@oozie.apache.org>" <de...@oozie.apache.org>>, Virag Kothari <vi...@yahoo-inc.com>>, Alejandro Abdelnur <tu...@cloudera.com>>
Subject: Re: OOZIE-1311

Virag,
I talked more with Alejandro about the sharelib and the launcher jar.

It would be good if we could eventually get rid of the launcher jar completely because:
- less files would be copied to HDFS so it should be faster
- it would fix an issue where some linux distress cleanup the temp dir and delete the launcher jar

The contents of the launcher jar could be moved to the Oozie sharelib (/share/lib/oozie) which IIRC currently only has a json jar file.

As for backwards compatibility, we could do something like what you suggested for the other sharelibs where the webapp module includes them.  Then we'd add some property to oozie-site that would switch between the old behavior with the launcher jar (sharelib is not required) and the new behavior without the launcher jar (sharelib is required).  A new major release, like Oozie 4, would be a good time to introduce this change as the default; users can use the property to go back to the old behavior.  Also, if set to use the new behavior, we should make oozie.use.system.libpath in the job.properties default to true because the sharelib would be required, and if set to the old behavior, it would default to false.

Thoughts?

thanks
- Robert





On Wed, Apr 24, 2013 at 12:24 PM, Virag Kothari <vi...@yahoo-inc.com>> wrote:
Thanks Robert. Created OOZIE-1341

On 4/24/13 11:36 AM, "Robert Kanter" <rk...@cloudera.com>> wrote:

>I think I follow what you're saying; assuming everything works correctly,
>I
>think that sounds good.  Can you make a JIRA with that and put up a patch;
>that might be easier to see how this would work.
>
>thanks
>- Robert
>
>
>On Wed, Apr 24, 2013 at 11:29 AM, Virag Kothari <vi...@yahoo-inc.com>>
>wrote:
>
>> We can keep the Main classes in the share lib, but have the action
>> executors package the main classes for shipping it to
>> Launcher (basically again having the method getLauncherClasses() in
>>action
>> executors). Also the oozie-sharelib.jar containing main classes
>> should be part of web-app.
>> This will improve the building/testing env as classes are in isolated
>> sharelibs but will make the share lib optional as main classes are part
>>of
>> oozie-server.
>> Can we do this change to trunk as this is blocking our QE's to pick
>>builds
>> due to share lib incompatibility?
>>
>> We can still do what oozie-1318 is trying to achieve. For, e.g with Hcat
>> integration,
>> there can be multiple overridable implementation of URIHandlers that can
>> be shipped to launcher.
>> We can do the same for other main classes if required.
>>
>>
>> Thanks,
>> Virag
>>
>>
>> On 4/24/13 10:18 AM, "Robert Kanter" <rk...@cloudera.com>> wrote:
>>
>> >That's true, its primarily a building/testing improvement.  And from a
>> >design perspective, it could be considered "cleaner" to not have the
>> >launcher jar.
>> >
>> >Your idea sounds like a good compromise between backwards compatibility
>> >and
>> >improving the building/testing env.  How would we do that?
>> >
>> >As a side note, there's also a new feature with the Main classes being
>>in
>> >the sharelib; users can now override the Main class that Oozie will use
>> >for
>> >the action because its no longer in the launcher jar.  I think this
>>could
>> >still be a useful feature, but I'm guessing that if we do the
>>compromise,
>> >this would go away?  In any case, OOZIE-1318 is to make the Main
>> >overridable with a config, which is probably cleaner and more flexible
>> >than
>> >replacing a jar file.
>> >
>> >thanks
>> >- Robert
>> >
>> >
>> >On Wed, Apr 24, 2013 at 9:54 AM, Virag Kothari <vi...@yahoo-inc.com>>
>> >wrote:
>> >
>> >> That is a good suggestion for preserving compatibility, Robert.
>> >>
>> >> Another question is what is the main purpose of OOZIE-1311? From JIRA
>> >> description, the primary
>> >> advantage of moving Main classes from core to share lib is it can
>>help
>> >>in
>> >> preventing dependencies conflicts
>> >> (different versions of antlr's for pig, hive). However, that problem
>> >>only
>> >> exists while building/testing Oozie.
>> >> If that is the case, main classes can be moved to share lib but they
>>can
>> >> be bundled in oozie-server itself.
>> >> In that case, the deployment doesn't change and share lib can still
>>be
>> >> optional
>> >> as oozie server would ship those classes (same as what was happening
>> >> before). And,
>> >> advantage of 1311 for building/testing will be retained
>> >>
>> >> The main concern is having share lib as a required installation will
>> >> complicate deployment.
>> >> Doing hot upgrade for share lib is non-trivial and Oozie doesn't
>>provide
>> >> inbuilt support for that.
>> >> Even though end-users wouldn't notice the change, it will be a big
>> >>change
>> >> for system admins installing oozie
>> >> So even though share lib is nice feature to have, keeping it optional
>> >> seems better to me as there are other ways
>> >> Of including jars and which were introduced much before share lib.
>> >>
>> >> Thanks,
>> >> Virag
>> >>
>> >>
>> >>
>> >>
>> >> On 4/24/13 9:09 AM, "Robert Kanter" <rk...@cloudera.com>> wrote:
>> >>
>> >> >Alejandro had another suggestion to prevent breaking stuff for
>>existing
>> >> >apps where the sharelib isn't used.  Assuming the sharelib is
>> >>installed,
>> >> >if
>> >> >the user doesn't set oozie.use.system.libpath to true, then Oozie
>>would
>> >> >only include the oozie-*.jar from the sharelib (the one with the
>>Main)
>> >>in
>> >> >it; if its set to true, then Oozie would include all of the jars in
>>the
>> >> >sharelib.  This would allow users to still use their own lib dir
>>while
>> >> >still getting the Main into the classpath.
>> >> >
>> >> >- Robert
>> >> >
>> >> >
>> >> >
>> >> >On Tue, Apr 23, 2013 at 10:46 PM, Alejandro Abdelnur
>> >> ><tu...@gmail.com>>wrote:
>> >> >
>> >> >> virag, if your oozie servers dont have sharelibs currently
>>installed
>> >> >>then
>> >> >> unstalling sharelibs as part of an update would not break ant
>>running
>> >> >>app.
>> >> >> regarding the requirement of enabling the sharelibs to have the
>> >>mains,
>> >> >>you
>> >> >> have a point. we need to figure out how to get those in the
>> >>classpath if
>> >> >> the user is not using the sharelibs. i'd suggest we leave this
>> >> >> patch in trunk for now and we move it to a release branch once we
>> >>sorted
>> >> >> out a way of solving this incompatibility.
>> >> >>
>> >> >> thanks
>> >> >>
>> >> >> Alejandro
>> >> >> (phone typing)
>> >> >>
>> >> >> On Apr 23, 2013, at 5:09 PM, Virag Kothari <vi...@yahoo-inc.com>>
>> >>wrote:
>> >> >>
>> >> >> >
>> >> >> > Tucu, Thanks for the reply and explanation.
>> >> >> > For workflows running in Y!, cold upgrade is not a feasible
>>option
>> >>as
>> >> >>we
>> >> >> > cannot expect all the running actions to complete before Oozie
>> >> >>restart.
>> >> >> > Hot upgrade is a good option, but we will need time to
>>implement it
>> >> >>as we
>> >> >> > currently don't have sharelib as part of our deployment.
>> >> >> > So can we deprecate the refactoring of launcher classes in this
>> >> >>release?
>> >> >> > Also, all the launcher classes are only added to distributed
>>cache
>> >>if
>> >> >>the
>> >> >> > workflow is configured to use system lib path
>> >> >> > (oozie.use.system.libpath=true)
>> >> >> > We shouldn't mandate the workflow to have this property as they
>>are
>> >> >> > oozie's launcher classes.
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Virag
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On 4/23/13 3:21 PM, "Alejandro Abdelnur" <tu...@gmail.com>>
>>wrote:
>> >> >> >
>> >> >> >> Virag,
>> >> >> >>
>> >> >> >> On sharelib being required, yes you are correct. For this we
>> >>should:
>> >> >> >>
>> >> >> >> * Make 100% clear in the quick-start/install docs that the
>> >>sharelib
>> >> >>is
>> >> >> >> REQUIRED.
>> >> >> >>
>> >> >> >> * Add a check at Oozie startup to verify the sharelib dir
>>exists,
>> >> >>else
>> >> >> >> fail
>> >> >> >> to start.
>> >> >> >>
>> >> >> >> On sharelib lib issue during upgrade for in-flight jobs.
>> >>Depending on
>> >> >> the
>> >> >> >> type of upgrade this may be an issue.
>> >> >> >>
>> >> >> >> * If you are upgrading an oozie server fix that does not change
>> >>the
>> >> >> >> sharelib files, this is not an issue and you can just shutdown
>>the
>> >> >>oozie
>> >> >> >> server with in-flight jobs.
>> >> >> >>
>> >> >> >> * If you are upgrading an oozie server fix that involves
>>sharelib
>> >> >>files
>> >> >> >> changes, then you have 2 options:
>> >> >> >>
>> >> >> >> ** Cold upgrade: bring all WF jobs to suspend/completion, wait
>> >>till
>> >> >>all
>> >> >> >> running actions end, then shutdown oozie server, upgrade oozie
>> >>server
>> >> >> and
>> >> >> >> sharelib. Then restart oozie server and resume WF jobs. In this
>> >>case
>> >> >>all
>> >> >> >> new WF actions will use the new sharelib.
>> >> >> >>
>> >> >> >> ** Hot upgrade: stop oozie server. modify the oozie-site.xml
>> >>sharelib
>> >> >> >> location to point to a new directory. upgrade the oozie server.
>> >> >>install
>> >> >> >> the
>> >> >> >> sharelib (will be a create as the sharelib dir in HDFS does not
>> >> >>exist).
>> >> >> >> start the oozie server. In this case all running WF actions
>>will
>> >> >> continue
>> >> >> >> running with no issues as the JARs in the distributed cache
>>have
>> >>not
>> >> >> been
>> >> >> >> touch. All new WF actions will start using the new sharelib.
>> >> >> >>
>> >> >> >> Note that this sharelib upgrade protocol is not introduced by
>> >> >>requiring
>> >> >> >> sharelib, it is required if you have applications that use
>> >>sharelib
>> >> >> today.
>> >> >> >>
>> >> >> >> Does this address your concerns?
>> >> >> >>
>> >> >> >> Thanks.
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> On Tue, Apr 23, 2013 at 1:35 PM, Virag Kothari
>> >><vi...@yahoo-inc.com>>
>> >> >> >> wrote:
>> >> >> >>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> With OOZIE-1311 and its subtasks,  the idea seems to move all
>>the
>> >> >> >>> launcher
>> >> >> >>> classes like PigMain, HiveMain etc. to  their respective
>> >>sharelibs.
>> >> >> >>> So, now shared lib is a mandatory deployment step. Before
>>shared
>> >>lib
>> >> >> was
>> >> >> >>> optional as users could bundle jars with their workflow
>> >>application.
>> >> >> >>> So always requiring shared lib seems to introduce 2 problems:
>> >> >> >>>
>> >> >> >>>  1.  The current deployments which don't use action shared lib
>> >>will
>> >> >> >>> fail.
>> >> >> >>> So, probably we should deprecate the current behavior.
>> >> >> >>>
>> >> >> >>> 2. The hadoop distributed cache mechanism will fail a job if
>>the
>> >> >>files
>> >> >> >>> in
>> >> >> >>> DC are updated on hdfs while the hadoop job is running. So,
>>when
>> >> >>Oozie
>> >> >> >>> is
>> >> >> >>> restarted and shared lib is uploaded to hdfs as part of
>> >> >> >>>             deployment, hadoop  will fail the existing jobs
>>for
>> >> >>which
>> >> >> >>> the
>> >> >> >>> timestamp of  the file on hdfs doesn't match the timestamp of
>>its
>> >> >>copy
>> >> >> >>> in
>> >> >> >>> the job's DC.
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> Thanks,
>> >> >> >>> Virag
>> >> >> >
>> >> >>
>> >>
>> >>
>>
>>