possible issues with multijdk support

[Sean Busbey <bu...@apache.org>]

I want to make sure I'm thinking about things correctly before filing
issues.

In HBase I'm setting up nightly jobs that rely on Yetus in QBR mode. On
some of our branches, we support both jdk7 and jdk8 so I want to run unit
tests for both. I'm currently using 0.5.0.

Here's the report from two example jobs (they link back to more jenkins
details):

https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1-HBASE-18147/1/Nightly_Build_Report/

https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1-HBASE-18147/2/Nightly_Build_Report/

Here's the pipeline they're referencing:

https://github.com/apache/hbase/blob/branch-1-HBASE-18147/dev-support/Jenkinsfile

In both of the above, jdk8 unit tests fail. One also has failures in jdk7.
Only the jdk7 status appears to be reported in the vote tables, and only
 jdk7 shows up in the "more details" section for the job where it fails.

AFAICT, the archiver functionality only runs after all the tests so it
fails to capture the surefire logs from the jkd8 run.

The job zips up the archiver directory after yetus runs (we got pinged for
using up gigs-per-run from test logs), so you have to download
test_logs.zip to see what the archiver did (I manually checked that the
directories match what was present prior to adding zipping htings up).

I think that the lack of a vote from jdk8 is a bug. I suspect the lack of
archiving the files that come out of multijdk builds other than the
JAVA_HOME one is just a feature gap (and would get filed as a proposed
improvement).

Does this all sound correct? Or have I botched configuration somehow?

Re: possible issues with multijdk support

[Sean Busbey <bu...@apache.org>]

Okay, so two different feature requests. :)


On Fri, Jul 14, 2017 at 2:51 PM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
>
>
> > On Jul 14, 2017, at 8:01 AM, Sean Busbey <bu...@apache.org> wrote:
> >
> > I think that the lack of a vote from jdk8 is a bug.
>
>       Well... maybe.  We've always only run one JDK during unit tests even in non-QBT mode due to the time restraints.  If we look at HBase's results,
> there are 2.5+hours of unit test time with just one JDK.  Running them both in succession leads to 5+hours.  For Hadoop, unit test time clocks in around
> 8-9 hours... for one JDK. Running two in succession would ... not be good.


For HBase this is would be perfect. I'd rather take up a single
executor for 5 hours than burn two executor for 2.5 hours each, if
only to keep more executors open for other projects to use. We only
need one set of results per day and we already try to allow jenkins to
schedule it via hash, so are relatively insensitive to how long the
job takes.

a run per JDK is a solid work around though, and I think I can tell
jenkins to not parallelize them. I'll just need to find a way to avoid
needing a second docker image.


On Fri, Jul 14, 2017 at 2:54 PM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
>
>
>
>         One other thing!
>
>         Keep in mind that Jenkins doesn't know that Yetus is running multiple JDKs.  So any reports (e.g., Test Trend report) that you would have it generate will still only show the last JDK result too.


Yep, but I'm expressly pointing jenkins to the test results within the
archiver output, so if there were results per jdk I could point it at
all of them.

Re: possible issues with multijdk support

[Allen Wittenauer <aw...@effectivemachines.com>]

	One other thing!

	Keep in mind that Jenkins doesn't know that Yetus is running multiple JDKs.  So any reports (e.g., Test Trend report) that you would have it generate will still only show the last JDK result too.  

Re: possible issues with multijdk support

[Allen Wittenauer <aw...@effectivemachines.com>]

> On Jul 14, 2017, at 8:01 AM, Sean Busbey <bu...@apache.org> wrote:
> 
> I want to make sure I'm thinking about things correctly before filing
> issues.
> 
> In HBase I'm setting up nightly jobs that rely on Yetus in QBR mode. On
> some of our branches, we support both jdk7 and jdk8 so I want to run unit
> tests for both. I'm currently using 0.5.0.

	It wasn't really written to handle multijdk mode at all.  The documentation doesn't really go into enough depth, and actually leads one to believe that it will work:

"It will preserve the directory structure of the source tree so that
multiple matching file names will be preserved."

	i.e., if there is foo/bar  and foo/foo/bar, you'll see both foo/bar and foo/foo/bar in your archive directory.  It does not mean ...

pass1:
	foo/bar
 	foo/foo/bar

pass2:
	foo/bar
	foo/foo/bar

...that'll you'll have two versions of foo/bar and foo/foo/bar for both passes.  That's essentially what multijdk mode is.  So, no, your findings aren't completely surprising to me.

> In both of the above, jdk8 unit tests fail. One also has failures in jdk7.
> Only the jdk7 status appears to be reported in the vote tables, and only
> jdk7 shows up in the "more details" section for the job where it fails.
> 
> AFAICT, the archiver functionality only runs after all the tests so it
> fails to capture the surefire logs from the jkd8 run.

	Exactly.  MultiJDK mode always tries to run the "lowest" version JDK last, so that's the version of the non-Yetus-controlled logs you'll get to keep.  e.g., diff-javadoc-javadoc-root-jdk1.7.0_131.txt and diff-javadoc-javadoc-root-jdk1.8.0_131.txt will be there, but target/blah/blah/blah will only have one set.

> I think that the lack of a vote from jdk8 is a bug.

	Well... maybe.  We've always only run one JDK during unit tests even in non-QBT mode due to the time restraints.  If we look at HBase's results, there are 2.5+hours of unit test time with just one JDK.  Running them both in succession leads to 5+hours.  For Hadoop, unit test time clocks in around 8-9 hours... for one JDK. Running two in succession would ... not be good.

> I suspect the lack of
> archiving the files that come out of multijdk builds other than the
> JAVA_HOME one is just a feature gap (and would get filed as a proposed
> improvement).
> 
> Does this all sound correct? Or have I botched configuration somehow?

	It all sounds like it is working as programmed. ;)

	In general, I'd recommend running *two* QBT jobs, one for each JDK.  This way you can run unit tests in parallel (the biggest time sync for large projects by far) as well as cut back on "extras".. i.e., checkstyle, findbugs, etc, probably only need to get run once.  Hadoop currently does this across platforms:  one QBT job for x86, one QBT job for PPC.  Since the PPC build is significantly slower, it only runs certain tests.

	The only bad part about this--and this could probably be worked around with a SMOP--is that it will generate two reports that need to get processed.