You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Hadoop QA (JIRA)" <ji...@apache.org> on 2015/10/02 08:35:26 UTC
[jira] [Commented] (OOZIE-2382)
org.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is
flakey
[ https://issues.apache.org/jira/browse/OOZIE-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940817#comment-14940817 ]
Hadoop QA commented on OOZIE-2382:
----------------------------------
Testing JIRA OOZIE-2382
Cleaning local git workspace
----------------------------
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
. {color:green}+1{color} the patch does not introduce any @author tags
. {color:green}+1{color} the patch does not introduce any tabs
. {color:green}+1{color} the patch does not introduce any trailing spaces
. {color:green}+1{color} the patch does not introduce any line longer than 132
. {color:green}+1{color} the patch does adds/modifies 3 testcase(s)
{color:green}+1 RAT{color}
. {color:green}+1{color} the patch does not seem to introduce new RAT warnings
{color:green}+1 JAVADOC{color}
. {color:green}+1{color} the patch does not seem to introduce new Javadoc warnings
{color:green}+1 COMPILE{color}
. {color:green}+1{color} HEAD compiles
. {color:green}+1{color} patch compiles
. {color:green}+1{color} the patch does not seem to introduce new javac warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
. {color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
. {color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
. {color:green}+1{color} distro tarball builds with the patch
----------------------------
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2567/
> org.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey
> -------------------------------------------------------------------------------
>
> Key: OOZIE-2382
> URL: https://issues.apache.org/jira/browse/OOZIE-2382
> Project: Oozie
> Issue Type: Bug
> Components: tests
> Affects Versions: trunk
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2382.001.patch
>
>
> {{rg.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID}} is flakey.
> {noformat}
> junit.framework.AssertionFailedError
> at junit.framework.Assert.fail(Assert.java:48)
> at junit.framework.Assert.assertTrue(Assert.java:20)
> at junit.framework.Assert.assertFalse(Assert.java:34)
> at junit.framework.Assert.assertFalse(Assert.java:41)
> at org.apache.oozie.action.hadoop.PigTestCase.testPig_withNullExternalID(PigTestCase.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We've seen this for months but has been really tricky to track down. The output of the test has this:
> {noformat}
> Run pig script using PigRunner.run() for Pig version 0.8+
> Apache Pig version 0.12.0-cdh5.5.0-SNAPSHOT (rexported)
> compiled Sep 30 2015, 22:11:56
> Hadoop Job IDs executed by Pig: job_20151001151153510_0001
> Run pig script using PigRunner.run() for Pig version 0.8+
> {noformat}
> Compared with a successful run of the test, the difference is that it shows a Hadoop Job ID. This test is supposed to make Pig fail and not run any jobs, which can be confirmed in the test output, yet there's a Hadoop Job ID here!
> It turns out that {{PigRunner}} eventually gets to {{PigStats}}, which is a singleton. And because we're running all of the Pig tests in the same JVM, {{PigStats}} is actually leaking through the tests. It doesn't seem to be a problem most of the time because it must be getting overwritten at each run, however, because {{testPig_withNullExternalID}} is purposefully failing the Pig job, it's not, and we end up with the {{PigStats}} from the previous test or a non-initialized {{PigStats}} depending on the order of the tests. If it's a previous {{PigStats}}, then {{PigMain}} will write the Hadoop Job ID to a new file for the {{testPig_withNullExternalID}} test, which will then fail because it's not expecting this file.
> Unfortunately, there isn't a clean way to reset {{PigStats}}. Pig v 0.9 has a {{set}} method which we can pass {{null}} to in order to reset the {{PigStats}}. However, this was changed in Pig v 0.13 to the {{start}} method. In either case, they're both package private, so we have to usereflection to find the existing method for whichever version of Pig we're using and make it public. We should do this after every Pig test, even if they don't all need it just to make sure that we're starting with a fresh Pig each time as unit tests are supposed to start fresh anyway.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)