[jira] [Created] (OOZIE-2382) rg.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey

["Robert Kanter (JIRA)" <ji...@apache.org>]

Robert Kanter created OOZIE-2382:
------------------------------------

             Summary: rg.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey
                 Key: OOZIE-2382
                 URL: https://issues.apache.org/jira/browse/OOZIE-2382
             Project: Oozie
          Issue Type: Bug
          Components: tests
    Affects Versions: trunk
            Reporter: Robert Kanter
            Assignee: Robert Kanter
             Fix For: trunk


{{rg.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID}} is flakey.  
{noformat}
junit.framework.AssertionFailedError
	at junit.framework.Assert.fail(Assert.java:48)
	at junit.framework.Assert.assertTrue(Assert.java:20)
	at junit.framework.Assert.assertFalse(Assert.java:34)
	at junit.framework.Assert.assertFalse(Assert.java:41)
	at org.apache.oozie.action.hadoop.PigTestCase.testPig_withNullExternalID(PigTestCase.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at junit.framework.TestCase.runTest(TestCase.java:168)
	at junit.framework.TestCase.runBare(TestCase.java:134)
	at junit.framework.TestResult$1.protect(TestResult.java:110)
	at junit.framework.TestResult.runProtected(TestResult.java:128)
	at junit.framework.TestResult.run(TestResult.java:113)
	at junit.framework.TestCase.run(TestCase.java:124)
	at junit.framework.TestSuite.runTest(TestSuite.java:243)
	at junit.framework.TestSuite.run(TestSuite.java:238)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:24)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
{noformat}

We've seen this for months but has been really tricky to track down.  The output of the test has this:
{noformat}
Run pig script using PigRunner.run() for Pig version 0.8+
Apache Pig version 0.12.0-cdh5.5.0-SNAPSHOT (rexported) 
compiled Sep 30 2015, 22:11:56
Hadoop Job IDs executed by Pig: job_20151001151153510_0001
Run pig script using PigRunner.run() for Pig version 0.8+
{noformat}
Compared with a successful run of the test, the difference is that it shows a Hadoop Job ID.  This test is supposed to make Pig fail and not run any jobs, which can be confirmed in the test output, yet there's a Hadoop Job ID here!

It turns out that {{PigRunner}} eventually gets to {{PigStats}}, which is a singleton.  And because we're running all of the Pig tests in the same JVM, {{PigStats}} is actually leaking through the tests.  It doesn't seem to be a problem most of the time because it must be getting overwritten at each run, however, because {{testPig_withNullExternalID}} is purposefully failing the Pig job, it's not, and we end up with the {{PigStats}} from the previous test or a non-initialized {{PigStats}} depending on the order of the tests.

Unfortunately, there isn't a clean way to reset {{PigStats}}.  Pig v 0.9 has a {{set}} method which we can pass {{null}} to in order to reset the {{PigStats}}.  However, this was changed in Pig v 0.13 to the {{start}} method.  In either case, they're both package private, so we have to usereflection to find the existing method for whichever version of Pig we're using and make it public.  We should do this after every Pig test, even if they don't all need it just to make sure that we're starting with a fresh Pig each time as unit tests are supposed to start fresh anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)