Testing oozie workflows

[Ed Kohlwey <ek...@gmail.com>]

Hi,
I'm trying to use the workflow integration testing facilities in Oozie.
I've done some minor hacking on them to get them to run outside of the
actual Oozie project - mostly changing some of the system properties so
that they no longer refer to the core maven project or any of the resources
in that project. I've developed a simple job that creates a two-column hsql
database in memory and executes a sqoop over the DB.

With the default memory configuration, the local "YarnNodeStatusUpdater" is
OOME'ing, citing maxPermGen space. The test hangs after that with a bunch
of garbage updates from various daemons.

I've set -Xmx to 1024m and -XX:maxPermGen to 512m, but that seems to make
the test just run for a really long time. I went up to 2 mins, seeing only
YARN container updates, and then thought I best e-mail the list to see if
there's any commonly encountered issues.

Has anyone encountered a similar issue with these tests? Should I expect +2
mins/test for something as sqooping from a local, in-memory DB? Or is this
unreasonable?

I'm running this on a 1 year-old-ish MBP with fairly-fast quad core
processor and 16 gb of ram.