You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Chris Lambertus <cm...@apache.org> on 2020/01/23 00:55:01 UTC
broken builds taking up resources
Folks,
Over the last week or so we have received many reports of broken builds due to nodes out of resources. As noted in INFRA-19751, builds appear to fail yet continue to run, using up all available resources on a build node.
I will be implementing a system to kill jenkins processes based on duration of run. My initial feeling is to kill any single process which has been running for longer than one hour real-time.
I will also be implementing a system to kill/purge all docker containers which have been running for over 6 hours.
I am seeking input on these time limits, especially from those with larger builds. Is there any reason a -single process- or a docker container should run for more than 1 or 6 hours respectively?
Thanks,
Chris
ASF Infra
Re: broken builds taking up resources
Posted by Josh Fischer <jo...@joshfischer.io>.
Hi,
The Heron project has a build that will last for about 2 hours and 40
minutes on average. It is a single Jenkins job that spins up two
different docker containers consecutively. We only run this job to
generate artifacts for a release. You can see the job here:
https://builds.apache.org/job/apache-heron-github-build-release-packages/lastBuild/
- Josh
On Wed, Jan 22, 2020 at 7:53 PM Chris Lambertus <cm...@apache.org> wrote:
>
>
> > On Jan 22, 2020, at 4:55 PM, Chris Lambertus <cm...@apache.org> wrote:
> >
> > Folks,
> >
> > Over the last week or so we have received many reports of broken builds
> due to nodes out of resources. As noted in INFRA-19751, builds appear to
> fail yet continue to run, using up all available resources on a build node.
> >
> > I will be implementing a system to kill jenkins processes based on
> duration of run. My initial feeling is to kill any single process which has
> been running for longer than one hour real-time.
> >
> > I will also be implementing a system to kill/purge all docker containers
> which have been running for over 6 hours.
>
> Additionally, orphaned docker jobs are causing major resource contention.
> I will be adding a weekly job to docker system prune —all && service docker
> restart.
>
> -Chris
>
>
>
> >
> >
> > I am seeking input on these time limits, especially from those with
> larger builds. Is there any reason a -single process- or a docker container
> should run for more than 1 or 6 hours respectively?
> >
> > Thanks,
> > Chris
> > ASF Infra
> >
>
>
Re: broken builds taking up resources
Posted by Joan Touzet <wo...@apache.org>.
On 2020-01-23 4:50, Chesnay Schepler wrote:
> On 23/01/2020 10:19, Thomas Bouron wrote:
>> On Thu, 23 Jan 2020 at 08:56, Robert Munteanu <ro...@apache.org> wrote:
>>
>>> On Wed, 2020-01-22 at 17:53 -0800, Chris Lambertus wrote:
>>>> Additionally, orphaned docker jobs are causing major resource
>>>> contention. I will be adding a weekly job to docker system prune —all
>>>> && service docker restart.
>>> +1, it's easy to get this wrong. It would be great if you could also
>>> document some best practices when using containers for CI on the ASF
>>> Jenkins instance.
>>>
>>> docker system prune -af works, but is probably a big hammer, when each
>>> job should instead clean up after itself.
>>>
>>>
>> +1 on this.
>>
>> For Brooklyn, we are using docker container for CI and I think we do it
>> right (you can see the Dockerfile and Jenkinsfile in the repo for more
>> context). But guidelines would be great. For instance, it took a long
>> time
>> to arrive to the particular setup we are currently running.
>>
>> Back to the question of timeouts, the main Brooklyn build pipeline takes
>> around 1h30.
>>
>> Best.
>>
> The Flink snapshot deployments take about 1h15m.
>
CouchDB is busy moving off the shared Jenkins workers, but our runs take
20-30 minutes on pull requests (unless there's a problem) and about
30-45 minutes on release branches after that. There are plans to extend
the release branch builds with other tasks that take longer, such as
preparing our Docker images -- these run rarely but do take upwards of
90-120 minutes each, and cannot be split into multiple steps.
-Joan
Re: broken builds taking up resources
Posted by Chesnay Schepler <ch...@apache.org>.
On 23/01/2020 10:19, Thomas Bouron wrote:
> On Thu, 23 Jan 2020 at 08:56, Robert Munteanu <ro...@apache.org> wrote:
>
>> On Wed, 2020-01-22 at 17:53 -0800, Chris Lambertus wrote:
>>> Additionally, orphaned docker jobs are causing major resource
>>> contention. I will be adding a weekly job to docker system prune —all
>>> && service docker restart.
>> +1, it's easy to get this wrong. It would be great if you could also
>> document some best practices when using containers for CI on the ASF
>> Jenkins instance.
>>
>> docker system prune -af works, but is probably a big hammer, when each
>> job should instead clean up after itself.
>>
>>
> +1 on this.
>
> For Brooklyn, we are using docker container for CI and I think we do it
> right (you can see the Dockerfile and Jenkinsfile in the repo for more
> context). But guidelines would be great. For instance, it took a long time
> to arrive to the particular setup we are currently running.
>
> Back to the question of timeouts, the main Brooklyn build pipeline takes
> around 1h30.
>
> Best.
>
The Flink snapshot deployments take about 1h15m.
Re: broken builds taking up resources
Posted by Thomas Bouron <th...@cloudsoft.io>.
On Thu, 23 Jan 2020 at 08:56, Robert Munteanu <ro...@apache.org> wrote:
> On Wed, 2020-01-22 at 17:53 -0800, Chris Lambertus wrote:
> > Additionally, orphaned docker jobs are causing major resource
> > contention. I will be adding a weekly job to docker system prune —all
> > && service docker restart.
>
> +1, it's easy to get this wrong. It would be great if you could also
> document some best practices when using containers for CI on the ASF
> Jenkins instance.
>
> docker system prune -af works, but is probably a big hammer, when each
> job should instead clean up after itself.
>
>
+1 on this.
For Brooklyn, we are using docker container for CI and I think we do it
right (you can see the Dockerfile and Jenkinsfile in the repo for more
context). But guidelines would be great. For instance, it took a long time
to arrive to the particular setup we are currently running.
Back to the question of timeouts, the main Brooklyn build pipeline takes
around 1h30.
Best.
--
Thomas Bouron
Senior Software Engineer
*Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
GitHub: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Re: broken builds taking up resources
Posted by Robert Munteanu <ro...@apache.org>.
On Wed, 2020-01-22 at 17:53 -0800, Chris Lambertus wrote:
> Additionally, orphaned docker jobs are causing major resource
> contention. I will be adding a weekly job to docker system prune —all
> && service docker restart.
+1, it's easy to get this wrong. It would be great if you could also
document some best practices when using containers for CI on the ASF
Jenkins instance.
docker system prune -af works, but is probably a big hammer, when each
job should instead clean up after itself.
Thanks,
Robert
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
> On Jan 22, 2020, at 4:55 PM, Chris Lambertus <cm...@apache.org> wrote:
>
> Folks,
>
> Over the last week or so we have received many reports of broken builds due to nodes out of resources. As noted in INFRA-19751, builds appear to fail yet continue to run, using up all available resources on a build node.
>
> I will be implementing a system to kill jenkins processes based on duration of run. My initial feeling is to kill any single process which has been running for longer than one hour real-time.
>
> I will also be implementing a system to kill/purge all docker containers which have been running for over 6 hours.
Additionally, orphaned docker jobs are causing major resource contention. I will be adding a weekly job to docker system prune —all && service docker restart.
-Chris
>
>
> I am seeking input on these time limits, especially from those with larger builds. Is there any reason a -single process- or a docker container should run for more than 1 or 6 hours respectively?
>
> Thanks,
> Chris
> ASF Infra
>
Re: broken builds taking up resources
Posted by Ismael Juma <is...@juma.me.uk>.
FYI, Apache Kafka builds take 3 to 4 hours to run currently (build, unit
and integration tests).
Thanks,
Ismael
On Wed, Jan 22, 2020 at 4:55 PM Chris Lambertus <cm...@apache.org> wrote:
> Folks,
>
> Over the last week or so we have received many reports of broken builds
> due to nodes out of resources. As noted in INFRA-19751, builds appear to
> fail yet continue to run, using up all available resources on a build node.
>
> I will be implementing a system to kill jenkins processes based on
> duration of run. My initial feeling is to kill any single process which has
> been running for longer than one hour real-time.
>
> I will also be implementing a system to kill/purge all docker containers
> which have been running for over 6 hours.
>
>
> I am seeking input on these time limits, especially from those with larger
> builds. Is there any reason a -single process- or a docker container should
> run for more than 1 or 6 hours respectively?
>
> Thanks,
> Chris
> ASF Infra
>
>
Re: broken builds taking up resources
Posted by Zoran Regvart <zo...@regvart.com>.
Hi Chris,
On Thu, Jan 23, 2020 at 1:55 AM Chris Lambertus <cm...@apache.org> wrote:
> I will be implementing a system to kill jenkins processes based on duration of run. My initial feeling is to kill any single process which has been running for longer than one hour real-time.
Can you provide some details on this. Camel builds take anywhere from
5-7 hours to complete, we have taken steps to reduce that by spinning
out components out of the camel main repository and investing into
build performance. For a project the size of Camel it's difficult to
confine the build within one hour.
zoran
--
Zoran Regvart
Re: broken builds taking up resources
Posted by Gavin McDonald <ip...@gmail.com>.
Hi,
On Sun, Jan 26, 2020 at 8:48 AM Mike Jumper <mj...@apache.org> wrote:
> It would be nice if Jenkins could be configured to recognize when a node is
> unusable due to lack of resources and automatically take it offline.
>
That would be a feature request with the people that write the Jenkins
software.
Gav...
>
> - Mike
>
> On Fri, Jan 24, 2020, 13:57 Chris Thistlethwaite <ch...@apache.org>
> wrote:
>
> > Here is some data from H41, which was rebooted last night and ran out of
> > threads today. https://paste.apache.org/lkmpq
> >
> > In this case it looks like Karaf was still stuck/broken even though
> > there were no builds running on H41 at the time I investigated.
> >
> > -Chris T.
> > #asfinfra
> >
> >
> > On 1/24/20 4:26 AM, Mick Semb Wever wrote:
> > >
> > >> Is there some way we can improve the visibility into disk usage on the
> > >> build nodes? How full they are? And what projects are taking up space?
> > >> Does jenkins provide this info? Or could infra dump a `du …` report
> > >> somewhere?
> > >
> > >
> > >
> > > There are two Jenkins plugins that help with this situation
> > > - CloudBees Disk Usage Simple
> > > - Percentage Disk Space Node Column
> > >
> > > Could either/both of them be an option to install?
> > >
> > > regards,
> > > Mick
> >
> >
>
--
Gav...
Re: broken builds taking up resources
Posted by Mike Jumper <mj...@apache.org>.
It would be nice if Jenkins could be configured to recognize when a node is
unusable due to lack of resources and automatically take it offline.
- Mike
On Fri, Jan 24, 2020, 13:57 Chris Thistlethwaite <ch...@apache.org> wrote:
> Here is some data from H41, which was rebooted last night and ran out of
> threads today. https://paste.apache.org/lkmpq
>
> In this case it looks like Karaf was still stuck/broken even though
> there were no builds running on H41 at the time I investigated.
>
> -Chris T.
> #asfinfra
>
>
> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
> >
> >> Is there some way we can improve the visibility into disk usage on the
> >> build nodes? How full they are? And what projects are taking up space?
> >> Does jenkins provide this info? Or could infra dump a `du …` report
> >> somewhere?
> >
> >
> >
> > There are two Jenkins plugins that help with this situation
> > - CloudBees Disk Usage Simple
> > - Percentage Disk Space Node Column
> >
> > Could either/both of them be an option to install?
> >
> > regards,
> > Mick
>
>
Re: broken builds taking up resources
Posted by Alexey Markevich <bu...@gmail.com>.
Hello,
I checked PR [1] and found many build failures on H41 during 'git'
command execution [2]. The last failure was OOM [3]. Finally build was
successful using H29 [4].
1. https://github.com/apache/cxf/pull/631
2. https://builds.apache.org/job/CXF-Trunk-PR/1389/console
3. https://builds.apache.org/job/CXF-Trunk-PR/1390/console
4. https://builds.apache.org/job/CXF-Trunk-PR/1391/
On 1/27/20, Chris Lambertus <cm...@apache.org> wrote:
> Here is some data from H24, which also contains many broken CXF jobs (not
> Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF
> builds, not karaf builds as previously noted. Copying dev@CXF since this
> build seems to be related to ongoing node problems.
>
>
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I
> have no idea what these are from, but they appear to be docker related.
>
>
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire
> 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp
> surefire_1029401344835483973390tmp
> jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire
> 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp
> surefire_359166059833874743459tmp
> jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45
> /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath
> /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf
> -Dmaven.home=/home/jenkins/tools/maven/latest3
> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14
> org.codehaus.plexus.classworlds.launcher.Launcher
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository
> -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep
> --color=auto CXF
>
>
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org>
>> wrote:
>>
>> Here is some data from H41, which was rebooted last night and ran out of
>> threads today. https://paste.apache.org/lkmpq
>>
>> In this case it looks like Karaf was still stuck/broken even though there
>> were no builds running on H41 at the time I investigated.
>>
>> -Chris T.
>> #asfinfra
>>
>>
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>>
>>>
>>> There are two Jenkins plugins that help with this situation
>>> - CloudBees Disk Usage Simple
>>> - Percentage Disk Space Node Column
>>>
>>> Could either/both of them be an option to install?
>>>
>>> regards,
>>> Mick
>>
>
>
Re: broken builds taking up resources
Posted by Alexey Markevich <bu...@gmail.com>.
Hello,
I checked PR [1] and found many build failures on H41 during 'git'
command execution [2]. The last failure was OOM [3]. Finally build was
successful using H29 [4].
1. https://github.com/apache/cxf/pull/631
2. https://builds.apache.org/job/CXF-Trunk-PR/1389/console
3. https://builds.apache.org/job/CXF-Trunk-PR/1390/console
4. https://builds.apache.org/job/CXF-Trunk-PR/1391/
On 1/27/20, Chris Lambertus <cm...@apache.org> wrote:
> Here is some data from H24, which also contains many broken CXF jobs (not
> Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF
> builds, not karaf builds as previously noted. Copying dev@CXF since this
> build seems to be related to ongoing node problems.
>
>
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I
> have no idea what these are from, but they appear to be docker related.
>
>
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire
> 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp
> surefire_1029401344835483973390tmp
> jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire
> 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp
> surefire_359166059833874743459tmp
> jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45
> /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath
> /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf
> -Dmaven.home=/home/jenkins/tools/maven/latest3
> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14
> org.codehaus.plexus.classworlds.launcher.Launcher
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository
> -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep
> --color=auto CXF
>
>
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org>
>> wrote:
>>
>> Here is some data from H41, which was rebooted last night and ran out of
>> threads today. https://paste.apache.org/lkmpq
>>
>> In this case it looks like Karaf was still stuck/broken even though there
>> were no builds running on H41 at the time I investigated.
>>
>> -Chris T.
>> #asfinfra
>>
>>
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>>
>>>
>>> There are two Jenkins plugins that help with this situation
>>> - CloudBees Disk Usage Simple
>>> - Percentage Disk Space Node Column
>>>
>>> Could either/both of them be an option to install?
>>>
>>> regards,
>>> Mick
>>
>
>
Re: broken builds taking up resources
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
>
> This is almost always because whatever is running on the two executors have suffocated the system resources.
... and before I forget, a reminder: Java threads take up a file descriptor. Hadoop's unit tests were firing up 10s of thousands of threads which were eating up 10s of thousands of FDs and ultimately lead to "cannot fork, no resource" errors causing everything to come tumbling down for the Jenkins salve process. So _all_ the resources, not just RAM or whatever.
Re: broken builds taking up resources
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
>
> This is almost always because whatever is running on the two executors have suffocated the system resources.
... and before I forget, a reminder: Java threads take up a file descriptor. Hadoop's unit tests were firing up 10s of thousands of threads which were eating up 10s of thousands of FDs and ultimately lead to "cannot fork, no resource" errors causing everything to come tumbling down for the Jenkins salve process. So _all_ the resources, not just RAM or whatever.
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
> On Jan 28, 2020, at 8:22 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
[snip]
> [1] - The best on-prem solution I came up with (before I moved my $DAYJOB stuff to cloud) was to run each executor in a VM on the box. That VM would also have a regularly scheduled job that would cause it to wipe itself and respawn via a trigger mechanism. Yeah, completely sucks, but at least it affords a lot more safety.
Allen,
I’d be interested in hearing more about your implementation here. Infra is moving towards a more compartmentalized strategy for jenkins, and I’m looking at how builds could be ephemeralized using k8s.
-Chris
Re: broken builds taking up resources
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jan 28, 2020, at 8:02 PM, Chris Lambertus <cm...@apache.org> wrote:
>
>
> Allen, can you elaborate on what a “proper” implementation is? As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.
>
You are correct: the limitations come from the implementation of the jenkins slave jar. Ideally it would run the slave.jar as one user and executors as one or more users. Or at least use cgroups on Linux and RBAC on Solaris and jails on FreeBSD and ... to at least do a minimal amount of work to protect itself. Instead, it depends upon the good will of spawned processes to not shoot it or anything else running on the box. This works great for the absolutely simple case, but completely false apart for anything beyond running a handful of shell commands.
Thus why I consider it idiotic. There are ways Jenkins could have done some work to prevent this situation from occurring, but alas that is not the case. Yes, it would require more setup of the client, but for those places that need (i.e., most) it would have been worth it.
Instead, on-prem operators are pretty much forced to build a ton of complex machinery to prevent users from wreaking havoc. [1] Or give up and move to either Jenkins talking to cloud or dump Jenkins entirely.
[1] - The best on-prem solution I came up with (before I moved my $DAYJOB stuff to cloud) was to run each executor in a VM on the box. That VM would also have a regularly scheduled job that would cause it to wipe itself and respawn via a trigger mechanism. Yeah, completely sucks, but at least it affords a lot more safety.
Re: broken builds taking up resources
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jan 28, 2020, at 8:02 PM, Chris Lambertus <cm...@apache.org> wrote:
>
>
> Allen, can you elaborate on what a “proper” implementation is? As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.
>
You are correct: the limitations come from the implementation of the jenkins slave jar. Ideally it would run the slave.jar as one user and executors as one or more users. Or at least use cgroups on Linux and RBAC on Solaris and jails on FreeBSD and ... to at least do a minimal amount of work to protect itself. Instead, it depends upon the good will of spawned processes to not shoot it or anything else running on the box. This works great for the absolutely simple case, but completely false apart for anything beyond running a handful of shell commands.
Thus why I consider it idiotic. There are ways Jenkins could have done some work to prevent this situation from occurring, but alas that is not the case. Yes, it would require more setup of the client, but for those places that need (i.e., most) it would have been worth it.
Instead, on-prem operators are pretty much forced to build a ton of complex machinery to prevent users from wreaking havoc. [1] Or give up and move to either Jenkins talking to cloud or dump Jenkins entirely.
[1] - The best on-prem solution I came up with (before I moved my $DAYJOB stuff to cloud) was to run each executor in a VM on the box. That VM would also have a regularly scheduled job that would cause it to wipe itself and respawn via a trigger mechanism. Yeah, completely sucks, but at least it affords a lot more safety.
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
>
>
>
>> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
>>
>> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
>> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
>> that could probably explain the hanging builds.
>
>
> This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect. (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed. This is an idiotic implementation, but it is what it is.)
this is correct re: starving the slave.jar. It’s entirely likely the CXF jobs are the ones causing the disconnect based on available information.
Allen, can you elaborate on what a “proper” implementation is? As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.
-Chris
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
>
>
>
>> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
>>
>> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
>> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
>> that could probably explain the hanging builds.
>
>
> This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect. (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed. This is an idiotic implementation, but it is what it is.)
this is correct re: starving the slave.jar. It’s entirely likely the CXF jobs are the ones causing the disconnect based on available information.
Allen, can you elaborate on what a “proper” implementation is? As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.
-Chris
Re: broken builds taking up resources
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
>
> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
> that could probably explain the hanging builds.
This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect. (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed. This is an idiotic implementation, but it is what it is.)
Anyway, in my experiences, if all/most of one type of job are failing with the node to appear to be crashed, then there is a good chance that job is the cause. So it would be great if someone could spend the effort to profile the CXF jobs to see what their actual resource consumption is.
FWIW, we had this problem with Hadoop, HBase, and others on the 'Hadoop' label nodes. The answer was to:
a) always run our jobs in containers that could be easily killed (freestyle Jenkins jobs that do 'docker run' generally can't be killed, despite what the UI says, because the signal never reaches the container)
b) those containers had resource limits
c) increase the resources that systemd is allowed to give the jenkins user
After doing that, the number of failures on the Hadoop nodes dropped exponentially.
Re: broken builds taking up resources
Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.
> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
>
> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
> that could probably explain the hanging builds.
This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect. (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed. This is an idiotic implementation, but it is what it is.)
Anyway, in my experiences, if all/most of one type of job are failing with the node to appear to be crashed, then there is a good chance that job is the cause. So it would be great if someone could spend the effort to profile the CXF jobs to see what their actual resource consumption is.
FWIW, we had this problem with Hadoop, HBase, and others on the 'Hadoop' label nodes. The answer was to:
a) always run our jobs in containers that could be easily killed (freestyle Jenkins jobs that do 'docker run' generally can't be killed, despite what the UI says, because the signal never reaches the container)
b) those containers had resource limits
c) increase the resources that systemd is allowed to give the jenkins user
After doing that, the number of failures on the Hadoop nodes dropped exponentially.
Re: broken builds taking up resources
Posted by Andriy Redko <dr...@gmail.com>.
Hi Chris,
Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
that could probably explain the hanging builds.
Best Regards,
Andriy Redko
CL> Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed
CL> because of broken container jobs from 2019, with over 1100(!) processes identical to:
CL> containerd-shim -namespace moby -workdir
CL> /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc
CL> -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
CL> /sigh
CL> asf999@asf940:~$ ps wwaxu | grep CXF
CL> jenkins 8380 0.0 0.5 39795216 541148 ? Sl Jan22 5:06
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
CL> jenkins 16834 0.0 0.5 39733804 496308 ? Sl Jan22 5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
CL> jenkins 25793 0.0 0.4 39797272 471988 ? Sl Jan22 5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
CL> jenkins 28293 0.0 0.5 39751220 521156 ? Sl Jan21 6:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
CL> jenkins 28431 0.1 1.2 6242916 1218032 ? Sl Jan23 6:35 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> jenkins 36663 0.0 0.5 39697712 556556 ? Sl Jan22 5:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
CL> jenkins 37958 0.0 0.5 4946924 567908 ? Sl Jan23 2:01 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> asf999 59270 0.0 0.0 16576 2088 pts/0 S+ 05:39 0:00 grep --color=auto CXF
>> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:
>> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
>> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
>> asf999@asf924:~$ ps wwaxu | grep CXF
>> jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
>> jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
>> jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
>> asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep --color=auto CXF
>>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>>> -Chris T.
>>> #asfinfra
>>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>>> Is there some way we can improve the visibility into disk usage on the
>>>>> build nodes? How full they are? And what projects are taking up space?
>>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>>> somewhere?
>>>> There are two Jenkins plugins that help with this situation
>>>> - CloudBees Disk Usage Simple
>>>> - Percentage Disk Space Node Column
>>>> Could either/both of them be an option to install?
>>>> regards,
>>>> Mick
Re: broken builds taking up resources
Posted by Andriy Redko <dr...@gmail.com>.
Hi Chris,
Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
that could probably explain the hanging builds.
Best Regards,
Andriy Redko
CL> Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed
CL> because of broken container jobs from 2019, with over 1100(!) processes identical to:
CL> containerd-shim -namespace moby -workdir
CL> /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc
CL> -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
CL> /sigh
CL> asf999@asf940:~$ ps wwaxu | grep CXF
CL> jenkins 8380 0.0 0.5 39795216 541148 ? Sl Jan22 5:06
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
CL> jenkins 16834 0.0 0.5 39733804 496308 ? Sl Jan22 5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
CL> jenkins 25793 0.0 0.4 39797272 471988 ? Sl Jan22 5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
CL> jenkins 28293 0.0 0.5 39751220 521156 ? Sl Jan21 6:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
CL> jenkins 28431 0.1 1.2 6242916 1218032 ? Sl Jan23 6:35 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> jenkins 36663 0.0 0.5 39697712 556556 ? Sl Jan22 5:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
CL> jenkins 37958 0.0 0.5 4946924 567908 ? Sl Jan23 2:01 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> asf999 59270 0.0 0.0 16576 2088 pts/0 S+ 05:39 0:00 grep --color=auto CXF
>> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:
>> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
>> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
>> asf999@asf924:~$ ps wwaxu | grep CXF
>> jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
>> jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
>> jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
>> asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep --color=auto CXF
>>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>>> -Chris T.
>>> #asfinfra
>>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>>> Is there some way we can improve the visibility into disk usage on the
>>>>> build nodes? How full they are? And what projects are taking up space?
>>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>>> somewhere?
>>>> There are two Jenkins plugins that help with this situation
>>>> - CloudBees Disk Usage Simple
>>>> - Percentage Disk Space Node Column
>>>> Could either/both of them be an option to install?
>>>> regards,
>>>> Mick
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed because of broken container jobs from 2019, with over 1100(!) processes identical to:
containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
/sigh
asf999@asf940:~$ ps wwaxu | grep CXF
jenkins 8380 0.0 0.5 39795216 541148 ? Sl Jan22 5:06 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
jenkins 16834 0.0 0.5 39733804 496308 ? Sl Jan22 5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
jenkins 25793 0.0 0.4 39797272 471988 ? Sl Jan22 5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
jenkins 28293 0.0 0.5 39751220 521156 ? Sl Jan21 6:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
jenkins 28431 0.1 1.2 6242916 1218032 ? Sl Jan23 6:35 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
jenkins 36663 0.0 0.5 39697712 556556 ? Sl Jan22 5:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
jenkins 37958 0.0 0.5 4946924 567908 ? Sl Jan23 2:01 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
asf999 59270 0.0 0.0 16576 2088 pts/0 S+ 05:39 0:00 grep --color=auto CXF
> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:
>
> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
>
>
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
>
>
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
> jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
> jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep --color=auto CXF
>
>
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>>
>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>>
>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>>
>> -Chris T.
>> #asfinfra
>>
>>
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>>
>>>
>>> There are two Jenkins plugins that help with this situation
>>> - CloudBees Disk Usage Simple
>>> - Percentage Disk Space Node Column
>>>
>>> Could either/both of them be an option to install?
>>>
>>> regards,
>>> Mick
>>
>
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed because of broken container jobs from 2019, with over 1100(!) processes identical to:
containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
/sigh
asf999@asf940:~$ ps wwaxu | grep CXF
jenkins 8380 0.0 0.5 39795216 541148 ? Sl Jan22 5:06 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
jenkins 16834 0.0 0.5 39733804 496308 ? Sl Jan22 5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
jenkins 25793 0.0 0.4 39797272 471988 ? Sl Jan22 5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
jenkins 28293 0.0 0.5 39751220 521156 ? Sl Jan21 6:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
jenkins 28431 0.1 1.2 6242916 1218032 ? Sl Jan23 6:35 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
jenkins 36663 0.0 0.5 39697712 556556 ? Sl Jan22 5:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
jenkins 37958 0.0 0.5 4946924 567908 ? Sl Jan23 2:01 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
asf999 59270 0.0 0.0 16576 2088 pts/0 S+ 05:39 0:00 grep --color=auto CXF
> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:
>
> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
>
>
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
>
>
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
> jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
> jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep --color=auto CXF
>
>
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>>
>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>>
>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>>
>> -Chris T.
>> #asfinfra
>>
>>
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>>
>>>
>>> There are two Jenkins plugins that help with this situation
>>> - CloudBees Disk Usage Simple
>>> - Percentage Disk Space Node Column
>>>
>>> Could either/both of them be an option to install?
>>>
>>> regards,
>>> Mick
>>
>
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
asf999@asf924:~$ ps wwaxu | grep CXF
jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep --color=auto CXF
> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>
> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>
> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>
> -Chris T.
> #asfinfra
>
>
> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>
>>> Is there some way we can improve the visibility into disk usage on the
>>> build nodes? How full they are? And what projects are taking up space?
>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>> somewhere?
>>
>>
>> There are two Jenkins plugins that help with this situation
>> - CloudBees Disk Usage Simple
>> - Percentage Disk Space Node Column
>>
>> Could either/both of them be an option to install?
>>
>> regards,
>> Mick
>
Re: broken builds taking up resources
Posted by Chris Lambertus <cm...@apache.org>.
Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
asf999@asf924:~$ ps wwaxu | grep CXF
jenkins 5951 0.0 0.8 40341652 813096 ? Sl Jan22 5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 6315 0.0 0.1 36400060 165960 ? Sl Jan22 4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
jenkins 6326 0.0 0.6 41248112 634760 ? Sl Jan22 6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 7527 0.1 0.3 38255856 383992 ? Sl Jan22 10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
jenkins 8669 0.0 0.6 40598416 635516 ? Sl Jan22 5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 15485 0.0 0.6 41069240 646600 ? Sl Jan22 6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 15535 0.0 0.4 39830952 427408 ? Sl Jan22 5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 20308 0.0 0.6 40012908 645164 ? Sl Jan22 5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 25889 0.0 0.9 40853360 977276 ? Sl Jan22 5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 30415 0.0 0.8 41621188 869660 ? Sl Jan22 6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins 30575 0.0 0.6 40199036 636260 ? Sl Jan22 5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins 31923 0.1 0.2 5073496 291664 ? Sl Jan26 0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
asf999 32332 0.0 0.0 16788 2112 pts/0 S+ 02:40 0:00 grep --color=auto CXF
> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>
> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>
> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>
> -Chris T.
> #asfinfra
>
>
> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>
>>> Is there some way we can improve the visibility into disk usage on the
>>> build nodes? How full they are? And what projects are taking up space?
>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>> somewhere?
>>
>>
>> There are two Jenkins plugins that help with this situation
>> - CloudBees Disk Usage Simple
>> - Percentage Disk Space Node Column
>>
>> Could either/both of them be an option to install?
>>
>> regards,
>> Mick
>
Re: broken builds taking up resources
Posted by Chris Thistlethwaite <ch...@apache.org>.
Here is some data from H41, which was rebooted last night and ran out of
threads today. https://paste.apache.org/lkmpq
In this case it looks like Karaf was still stuck/broken even though
there were no builds running on H41 at the time I investigated.
-Chris T.
#asfinfra
On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>
>> Is there some way we can improve the visibility into disk usage on the
>> build nodes? How full they are? And what projects are taking up space?
>> Does jenkins provide this info? Or could infra dump a `du …` report
>> somewhere?
>
>
>
> There are two Jenkins plugins that help with this situation
> - CloudBees Disk Usage Simple
> - Percentage Disk Space Node Column
>
> Could either/both of them be an option to install?
>
> regards,
> Mick
Re: broken builds taking up resources
Posted by Mick Semb Wever <mc...@apache.org>.
> Is there some way we can improve the visibility into disk usage on the
> build nodes? How full they are? And what projects are taking up space?
> Does jenkins provide this info? Or could infra dump a `du …` report
> somewhere?
There are two Jenkins plugins that help with this situation
- CloudBees Disk Usage Simple
- Percentage Disk Space Node Column
Could either/both of them be an option to install?
regards,
Mick
Re: broken builds taking up resources
Posted by Mick Semb Wever <mc...@apache.org>.
The Cassandra dtest builds take ~12 hours. The unit tests over an hour
We are looking into parallelising these, but work hasn't started on that yet.
We recently parallelised a number of the unit test builds, and added pipeline builds, and subsequently builds have been crashing with full disks. Yesterday it was cassandra12 and cassandra15. I wiped their Cassandra build workspaces and they are running again. I also reduce the log retention rate from 50 to 25 builds (which I didn't want to do, we'd rather kept more if we knew we could).
But if this was from other projects it would be great to know.
Is there some way we can improve the visibility into disk usage on the build nodes? How full they are? And what projects are taking up space? Does jenkins provide this info? Or could infra dump a `du …` report somewhere?
regards,
Mick
On Thu, 23 Jan 2020, at 07:26, Martin Stockhammer wrote:
> Hi,
>
> our average build time for the main archiva build job is about 1 hour on
> the apache build servers.
> We have a timeout of 2h configured in our pipeline.
>
> So, one hour is too short for us and we would appreciate, if you
> consider to increase your kill timeout to some higher value.
>
> Regards
>
> Martin
>
>
> On 23.01.20 01:55, Chris Lambertus wrote:
> > Folks,
> >
> > Over the last week or so we have received many reports of broken builds due to nodes out of resources. As noted in INFRA-19751, builds appear to fail yet continue to run, using up all available resources on a build node.
> >
> > I will be implementing a system to kill jenkins processes based on duration of run. My initial feeling is to kill any single process which has been running for longer than one hour real-time.
> >
> > I will also be implementing a system to kill/purge all docker containers which have been running for over 6 hours.
> >
> >
> > I am seeking input on these time limits, especially from those with larger builds. Is there any reason a -single process- or a docker container should run for more than 1 or 6 hours respectively?
> >
> > Thanks,
> > Chris
> > ASF Infra
> >
> >
>
Re: broken builds taking up resources
Posted by Martin Stockhammer <ma...@apache.org>.
Hi,
our average build time for the main archiva build job is about 1 hour on
the apache build servers.
We have a timeout of 2h configured in our pipeline.
So, one hour is too short for us and we would appreciate, if you
consider to increase your kill timeout to some higher value.
Regards
Martin
On 23.01.20 01:55, Chris Lambertus wrote:
> Folks,
>
> Over the last week or so we have received many reports of broken builds due to nodes out of resources. As noted in INFRA-19751, builds appear to fail yet continue to run, using up all available resources on a build node.
>
> I will be implementing a system to kill jenkins processes based on duration of run. My initial feeling is to kill any single process which has been running for longer than one hour real-time.
>
> I will also be implementing a system to kill/purge all docker containers which have been running for over 6 hours.
>
>
> I am seeking input on these time limits, especially from those with larger builds. Is there any reason a -single process- or a docker container should run for more than 1 or 6 hours respectively?
>
> Thanks,
> Chris
> ASF Infra
>
>