[jira] [Commented] (HADOOP-13514) Upgrade maven surefire plugin to 2.19.1

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16236972#comment-16236972 ] 

Allen Wittenauer commented on HADOOP-13514:
-------------------------------------------

Latest version of surefire is 2.20.1, released in September.

I've been spending a few weeks looking at our usage on the ASF build machines.  We typically have anywhere from 8 to 40 zombie JVMs running around after the hadoop-common, hadoop-hdfs, and hadoop-mapreduce allegedly complete. These do nothing but eat resources. As a result, timeouts are pretty normal and problems that are claimed to be "environmental" are mostly self-inflicted by badly written tests that really only get any stress on the build machines.

I'm fairly convinced that HDFS-12711 can be prevented with SUREFIRE-773, which was closed as a dupe of SUREFIRE-524.  At this point, I think we absolutely need to make this a priority to get into at least trunk and branch-2 ASAP.  

[It's easy to say... "this wasn't a problem before!"... I'm not so convinced it wasn't.  I think we just didn't see it as often.  But new tests are doing new things that eat more resources and stay around longer which impacts more runs.  Additionally, running unit tests under Docker guaranteed that when the container died, so did all of these stale JVMs.  ]


> Upgrade maven surefire plugin to 2.19.1
> ---------------------------------------
>
>                 Key: HADOOP-13514
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13514
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 2.8.0
>            Reporter: Ewan Higgs
>            Assignee: Akira Ajisaka
>            Priority: Major
>         Attachments: HADOOP-13514-addendum.01.patch, HADOOP-13514-testing.001.patch, HADOOP-13514-testing.002.patch, HADOOP-13514-testing.003.patch, HADOOP-13514-testing.004.patch, HADOOP-13514.002.patch, HADOOP-13514.003.patch, HADOOP-13514.004.patch, HADOOP-13514.005.patch, surefire-2.19.patch
>
>
> A lot of people working on Hadoop don't want to run all the tests when they develop; only the bits they're working on. Surefire 2.19 introduced more useful test filters which let us run a subset of the tests that brings the build time down from 'come back tomorrow' to 'grab a coffee'.
> For instance, if I only care about the S3 adaptor, I might run:
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*, org.apache.hadoop.fs.s3a.*\"
> {code}
> We can work around this by specifying the surefire version on the command line but it would be better, imo, to just update the default surefire used.
> {code}
> mvn test -Dmaven.javadoc.skip=true -Pdist,native -Djava.awt.headless=true \"-Dtest=org.apache.hadoop.fs.*, org.apache.hadoop.hdfs.*, org.apache.hadoop.fs.s3a.*\" -Dmaven-surefire-plugin.version=2.19.1
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org