You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Alex Newman <po...@gmail.com> on 2014/08/13 02:30:39 UTC
Jenkins jobs leave junk around
So I am trying to setup an internal jenkins. I have noticed, once
builds complete (or fail), sometimes we have processes that stay
around
20242 ? Sl 3:27 /usr/java/jdk1.7.0_55/jre/bin/java
-enableassertions -XX:MaxDirectMemorySize=1G -Xmx1900m
-XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
/var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefirebooter1201010766180838953.jar
/var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire6309574441534864799tmp
/var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire_2192766844117343368134tmp
Our pre build maven step uses
-DcompileSource=1.7 clean install -DskipTests to prepare the directory
Then we run
-PrunAllTests -DcompileSource=1.7
-Dmaven.test.redirectTestOutputToFile=true install
-Dsurefire.secondPartThreadCount=5 -Dit.test=noItTest
I am curious if this is a common problem. Any tricks we can do to speed this up?
-Alex Newman
Re: Jenkins jobs leave junk around
Posted by Ted Yu <yu...@gmail.com>.
Apache build uses the following script to capture jstack of the hanging
test:
ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
#It seems sometimes the tests are not dying immediately. Let's give
them 10s
echo "Suspicious java process found - waiting 10s to see if there are
just slow to stop"
sleep 10
ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
echo "There are $ZOMBIE_TESTS_COUNT zombie tests, they should have
been killed by surefire but survived"
echo "************ BEGIN zombies jstack extract"
ZB_STACK=`jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1
jstack | grep ".test" | grep "\.java"`
jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack
echo "************ END zombies jstack extract"
JIRA_COMMENT="$JIRA_COMMENT
{color:red}-1 core zombie tests{color}. There are
${ZOMBIE_TESTS_COUNT} zombie test(s): ${ZB_STACK}"
BAD=1
jps | grep surefirebooter | cut -d ' ' -f 1 | xargs kill -9
else
echo "We're ok: there is no zombie test, but some tests took some
time to stop"
fi
else
echo "We're ok: there is no zombie test"
fi
On Tue, Aug 12, 2014 at 6:02 PM, Mikhail Antonov <ol...@gmail.com>
wrote:
> I assume that as a quick-fix solution it's safe to kill those
> processes in the post-build jenkins steps.
>
> -Mikhail
>
> 2014-08-12 18:00 GMT-07:00, Ted Yu <yu...@gmail.com>:
> > Apache Jenkins sets surefire.secondPartThreadCount at 2 - so your build
> > should take shorter than Apache trunk build.
> >
> > Cheers
> >
> >
> > On Tue, Aug 12, 2014 at 5:30 PM, Alex Newman <po...@gmail.com> wrote:
> >
> >> So I am trying to setup an internal jenkins. I have noticed, once
> >> builds complete (or fail), sometimes we have processes that stay
> >> around
> >>
> >> 20242 ? Sl 3:27 /usr/java/jdk1.7.0_55/jre/bin/java
> >> -enableassertions -XX:MaxDirectMemorySize=1G -Xmx1900m
> >> -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> >> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
> >>
> >>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefirebooter1201010766180838953.jar
> >>
> >>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire6309574441534864799tmp
> >>
> >>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire_2192766844117343368134tmp
> >>
> >>
> >> Our pre build maven step uses
> >> -DcompileSource=1.7 clean install -DskipTests to prepare the directory
> >>
> >> Then we run
> >> -PrunAllTests -DcompileSource=1.7
> >> -Dmaven.test.redirectTestOutputToFile=true install
> >> -Dsurefire.secondPartThreadCount=5 -Dit.test=noItTest
> >>
> >> I am curious if this is a common problem. Any tricks we can do to speed
> >> this up?
> >>
> >> -Alex Newman
> >>
> >
>
>
> --
> Thanks,
> Michael Antonov
>
Re: Jenkins jobs leave junk around
Posted by Mikhail Antonov <ol...@gmail.com>.
I assume that as a quick-fix solution it's safe to kill those
processes in the post-build jenkins steps.
-Mikhail
2014-08-12 18:00 GMT-07:00, Ted Yu <yu...@gmail.com>:
> Apache Jenkins sets surefire.secondPartThreadCount at 2 - so your build
> should take shorter than Apache trunk build.
>
> Cheers
>
>
> On Tue, Aug 12, 2014 at 5:30 PM, Alex Newman <po...@gmail.com> wrote:
>
>> So I am trying to setup an internal jenkins. I have noticed, once
>> builds complete (or fail), sometimes we have processes that stay
>> around
>>
>> 20242 ? Sl 3:27 /usr/java/jdk1.7.0_55/jre/bin/java
>> -enableassertions -XX:MaxDirectMemorySize=1G -Xmx1900m
>> -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
>> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
>>
>> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefirebooter1201010766180838953.jar
>>
>> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire6309574441534864799tmp
>>
>> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire_2192766844117343368134tmp
>>
>>
>> Our pre build maven step uses
>> -DcompileSource=1.7 clean install -DskipTests to prepare the directory
>>
>> Then we run
>> -PrunAllTests -DcompileSource=1.7
>> -Dmaven.test.redirectTestOutputToFile=true install
>> -Dsurefire.secondPartThreadCount=5 -Dit.test=noItTest
>>
>> I am curious if this is a common problem. Any tricks we can do to speed
>> this up?
>>
>> -Alex Newman
>>
>
--
Thanks,
Michael Antonov
Re: Jenkins jobs leave junk around
Posted by Ted Yu <yu...@gmail.com>.
Apache Jenkins sets surefire.secondPartThreadCount at 2 - so your build
should take shorter than Apache trunk build.
Cheers
On Tue, Aug 12, 2014 at 5:30 PM, Alex Newman <po...@gmail.com> wrote:
> So I am trying to setup an internal jenkins. I have noticed, once
> builds complete (or fail), sometimes we have processes that stay
> around
>
> 20242 ? Sl 3:27 /usr/java/jdk1.7.0_55/jre/bin/java
> -enableassertions -XX:MaxDirectMemorySize=1G -Xmx1900m
> -XX:MaxPermSize=256m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefirebooter1201010766180838953.jar
>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire6309574441534864799tmp
>
> /var/lib/jenkins/workspace/hbase_asf_master_box1/hbase-server/target/surefire/surefire_2192766844117343368134tmp
>
>
> Our pre build maven step uses
> -DcompileSource=1.7 clean install -DskipTests to prepare the directory
>
> Then we run
> -PrunAllTests -DcompileSource=1.7
> -Dmaven.test.redirectTestOutputToFile=true install
> -Dsurefire.secondPartThreadCount=5 -Dit.test=noItTest
>
> I am curious if this is a common problem. Any tricks we can do to speed
> this up?
>
> -Alex Newman
>