You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Robert Kanter (JIRA)" <ji...@apache.org> on 2014/11/18 22:57:36 UTC

[jira] [Updated] (OOZIE-1770) Create Oozie Application Master for YARN

     [ https://issues.apache.org/jira/browse/OOZIE-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Kanter updated OOZIE-1770:
---------------------------------
    Attachment: oya.patch
                oya-rm-screenshot.jpg

We did the code against CDH, but I was able to port it back over to Apache trunk without conflicts other than pom changes.  I decided that it was better to make this available sooner rather than spend too much time spiffying it up.

Here’s a brief overview of the design:
Oozie has an unmanaged AM “pool” that it uses for submitting jobs.  We need a pool because we have to create an AM for each user that submits a job (we adapted some code from Llama).  When Oozie wants to submit a job, instead of submitting an MR launcher job, it can create/get one of these AM’s and use it to create a Yarn container, and then run the launcher in that container.  During our testing, we were using a Java action that launches a simple MR job.  In the screenshot below, you can see that we have the one “OozieServer” AM, and then 3 MAPREDUCE applications, from when we ran the workflow 3 times.  The OozieServer AM was reused each time to submit the MR jobs, and there’s no longer a Launcher Job.
!oya-rm-screenshot.jpg!

Given that this was more of a proof-of-concept and we didn’t have a lot of time, we didn’t redo the launcher code.  It still uses LauncherMapper; I just hacked in some extra methods for running it outside of a map task so we could run it in the container.  This is definitely an area where we can improve things a lot.  One major thing to keep in mind is that the container gives us a Shell; right now, we’re then starting a JVM to run the LauncherMapper code, but it probably would make sense to see if we can skip the JVM and run most actions directly in the shell.

Interestingly, running a Container that doesn’t do much (e.g. a “Hello World” Java action), runs so fast, that Oozie is now the bottleneck.  The callback comes in before the action has a chance to transition to RUNNING, so Oozie complains.  We fixed this by adding a delay.  We’ll probably want to improve this.

I had to remove MR1 support, obviously.  This greatly simplifies the build because we don’t need hadooplibs anymore.  I also had to up the Hadoop version to 2.4.0+ from 2.3.0.

We can probably get rid of this eventually, but for now, I made it so that if you set {{ooze.use.jobclient.launch=true}} in oozie-site, Oozie will use the old MR launcher job behavior instead of the container behavior.

Here’s a to do list of things that need to be fixed or improved:
- The container currently runs as the Yarn user regardless of who submitted the workflow.  So, permissions-wise, workflows only work if you submit them as the Yarn user.  
- The shell action should not start a JVM
- Actions should be refactored/cleaned-up/simplified to not need all the stuff LauncherMapper is doing
	-- Regardless of whether or not we still start a JVM, there’s a bunch of stuff LauncherMapper does that we should get rid of or simplify or change
	-- The Shell action definitely doesn’t need a JVM; once the user problem is fixed, it should also finally run as the proper user!
- The NMToken expires after 10-15min and you can’t submit containers anymore from the AM’s in the AM pool
- The Oozie kill command still expects an MR job and doesn’t work; it needs to use the container id
- The status checking code is pretty hacky.  I modified it to check the container status, but it doesn’t check if the job actually failed or not
- The callback is pretty hacky.  The Launcher’s MR AM was sending the callback to Oozie before.  I had to make the container do this now
	-- It may make sense to come up with a different mechanism for this
- I haven’t tried any recovery stuff or Oozie HA
- I had to add two columns: one to store node Id host and one to store node Id port.  OozieDBCLI needs to be updated to create these during an upgrade.  Creating a new database works fine though because OpenJPA handles it.
- We’ve only tested with the Java action.  Most of the other actions should work with some minor tweaking (need to set expected extra env vars, etc).  The MR action probably won’t work because of the swapping optimization; it probably makes sense to get rid of that.

The oya.patch has all of the changes.

> Create Oozie Application Master for YARN
> ----------------------------------------
>
>                 Key: OOZIE-1770
>                 URL: https://issues.apache.org/jira/browse/OOZIE-1770
>             Project: Oozie
>          Issue Type: New Feature
>            Reporter: Bowen Zhang
>            Assignee: Bowen Zhang
>         Attachments: oya-rm-screenshot.jpg, oya.patch
>
>
> After the first release of oozie on hadoop 2, it will be good if users can set execution engine in oozie conf, be it YARN AM or traditional MR. We can target this for post oozie 4.1 release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)