maven-surefire-plugin configuration for Hadoop 2.8.5

[Elaine Ang <el...@gmail.com>]

Hi all,

I'm running unit tests for Hadoop 2.8.5. I had a script that basically
repeatedly run command
   mvn -Dtest=t1,t2,t3,... test
with different test selections. And after each test command finishes, I
need to get test running stats for other purposes.

The problem is that I saw lingering java processes running
surefirebooterxxxxx.jar that consumes a lot of memory and CPU resources. Is
this owing to setting  <forkedProcessTimeoutInSeconds> to a timeout value
under maven-surefire-plugin configuration (in hadoop-project/pom.xml)?

If so, I'm wondering if there are specific reasons (e.g. required by
certain unit tests) for using a timeout value and delay the killing of
forked JVMs, instead of doing something like <shutdown>exit</shutdown>?
I changed my pom configuration to just using shutdown exit and I didn't see
lingering java surefirebooter processes anymore, but I'm not sure if this
change would break any Hadoop unit test.

Any thoughts on this? Thanks a lot in advance.

Best,
Elaine

Re: maven-surefire-plugin configuration for Hadoop 2.8.5

[Steve Loughran <st...@cloudera.com.INVALID>]

On Wed, May 1, 2019 at 6:29 PM Elaine Ang <el...@gmail.com> wrote:

> Hi all,
>
> I'm running unit tests for Hadoop 2.8.5. I had a script that basically
> repeatedly run command
>    mvn -Dtest=t1,t2,t3,... test
> with different test selections. And after each test command finishes, I
> need to get test running stats for other purposes.
>
> The problem is that I saw lingering java processes running
> surefirebooterxxxxx.jar that consumes a lot of memory and CPU resources. Is
> this owing to setting  <forkedProcessTimeoutInSeconds> to a timeout value
> under maven-surefire-plugin configuration (in hadoop-project/pom.xml)?
>

the timeout is there to kill hung tests so that jenkins doesn't get blocked.


>
> If so, I'm wondering if there are specific reasons (e.g. required by
> certain unit tests) for using a timeout value and delay the killing of
> forked JVMs, instead of doing something like <shutdown>exit</shutdown>?
> I changed my pom configuration to just using shutdown exit and I didn't see
> lingering java surefirebooter processes anymore, but I'm not sure if this
> change would break any Hadoop unit test.
>
> Looking at the maven docs, the shutdown operation seems to be for handling
the explicit kill of the maven process. I don't see any reason not to add
that, so please file a JIRA and then link to a github PR for the change.

If you really are doing complex test work, then the limitations of the
maven test runner (it's hard-coded generation of the XML DOM test report
originally written for Ant's <junitreport> surface: it builds up the entire
DOM in memory and only saves it when the test suites finish, just so that
the summary xml attributes can be added. You are free to implement your own
JUnit test runner which generates different reports (indeed, I Ant's
<junit> runnier will let you plug in different listeners to build different
reports). If you find yourself waiting for a full surefire test run to
complete before getting the results, know that you have the option not to
do that.

-Steve




> Any thoughts on this? Thanks a lot in advance.
>
> Best,
> Elaine
>

Re: maven-surefire-plugin configuration for Hadoop 2.8.5

[Steve Loughran <st...@cloudera.com.INVALID>]

On Wed, May 1, 2019 at 6:29 PM Elaine Ang <el...@gmail.com> wrote:

> Hi all,
>
> I'm running unit tests for Hadoop 2.8.5. I had a script that basically
> repeatedly run command
>    mvn -Dtest=t1,t2,t3,... test
> with different test selections. And after each test command finishes, I
> need to get test running stats for other purposes.
>
> The problem is that I saw lingering java processes running
> surefirebooterxxxxx.jar that consumes a lot of memory and CPU resources. Is
> this owing to setting  <forkedProcessTimeoutInSeconds> to a timeout value
> under maven-surefire-plugin configuration (in hadoop-project/pom.xml)?
>

the timeout is there to kill hung tests so that jenkins doesn't get blocked.


>
> If so, I'm wondering if there are specific reasons (e.g. required by
> certain unit tests) for using a timeout value and delay the killing of
> forked JVMs, instead of doing something like <shutdown>exit</shutdown>?
> I changed my pom configuration to just using shutdown exit and I didn't see
> lingering java surefirebooter processes anymore, but I'm not sure if this
> change would break any Hadoop unit test.
>
> Looking at the maven docs, the shutdown operation seems to be for handling
the explicit kill of the maven process. I don't see any reason not to add
that, so please file a JIRA and then link to a github PR for the change.

If you really are doing complex test work, then the limitations of the
maven test runner (it's hard-coded generation of the XML DOM test report
originally written for Ant's <junitreport> surface: it builds up the entire
DOM in memory and only saves it when the test suites finish, just so that
the summary xml attributes can be added. You are free to implement your own
JUnit test runner which generates different reports (indeed, I Ant's
<junit> runnier will let you plug in different listeners to build different
reports). If you find yourself waiting for a full surefire test run to
complete before getting the results, know that you have the option not to
do that.

-Steve




> Any thoughts on this? Thanks a lot in advance.
>
> Best,
> Elaine
>