Re: broken builds taking up resources

[Chris Lambertus <cm...@apache.org>]

Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.


Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.


asf999@asf924:~$ ps wwaxu | grep CXF
jenkins   5951  0.0  0.8 40341652 813096 ?     Sl   Jan22   5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins   6315  0.0  0.1 36400060 165960 ?     Sl   Jan22   4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
jenkins   6326  0.0  0.6 41248112 634760 ?     Sl   Jan22   6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins   7527  0.1  0.3 38255856 383992 ?     Sl   Jan22  10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
jenkins   8669  0.0  0.6 40598416 635516 ?     Sl   Jan22   5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins  15485  0.0  0.6 41069240 646600 ?     Sl   Jan22   6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins  15535  0.0  0.4 39830952 427408 ?     Sl   Jan22   5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins  20308  0.0  0.6 40012908 645164 ?     Sl   Jan22   5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins  25889  0.0  0.9 40853360 977276 ?     Sl   Jan22   5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins  30415  0.0  0.8 41621188 869660 ?     Sl   Jan22   6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
jenkins  30575  0.0  0.6 40199036 636260 ?     Sl   Jan22   5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
jenkins  31923  0.1  0.2 5073496 291664 ?      Sl   Jan26   0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
asf999   32332  0.0  0.0  16788  2112 pts/0    S+   02:40   0:00 grep --color=auto CXF


> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
> 
> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
> 
> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
> 
> -Chris T.
> #asfinfra
> 
> 
> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>> 
>>> Is there some way we can improve the visibility into disk usage on the
>>> build nodes? How full they are? And what projects are taking up space?
>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>> somewhere?
>>  
>> 
>> There are two Jenkins plugins that help with this situation
>>  - CloudBees Disk Usage Simple
>>  - Percentage Disk Space Node Column
>> 
>> Could either/both of them be an option to install?
>> 
>> regards,
>> Mick
> 

Re: broken builds taking up resources

[Alexey Markevich <bu...@gmail.com>]

Hello,

I checked PR [1] and found many build failures on H41 during 'git'
command execution [2]. The last failure was OOM [3]. Finally build was
successful using H29 [4].

1. https://github.com/apache/cxf/pull/631
2. https://builds.apache.org/job/CXF-Trunk-PR/1389/console
3. https://builds.apache.org/job/CXF-Trunk-PR/1390/console
4. https://builds.apache.org/job/CXF-Trunk-PR/1391/

On 1/27/20, Chris Lambertus <cm...@apache.org> wrote:
> Here is some data from H24, which also contains many broken CXF jobs (not
> Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF
> builds, not karaf builds as previously noted. Copying dev@CXF since this
> build seems to be related to ongoing node problems.
>
>
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I
> have no idea what these are from, but they appear to be docker related.
>
>
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins   5951  0.0  0.8 40341652 813096 ?     Sl   Jan22   5:45
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins   6315  0.0  0.1 36400060 165960 ?     Sl   Jan22   4:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire
> 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp
> surefire_1029401344835483973390tmp
> jenkins   6326  0.0  0.6 41248112 634760 ?     Sl   Jan22   6:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins   7527  0.1  0.3 38255856 383992 ?     Sl   Jan22  10:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire
> 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp
> surefire_359166059833874743459tmp
> jenkins   8669  0.0  0.6 40598416 635516 ?     Sl   Jan22   5:52
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  15485  0.0  0.6 41069240 646600 ?     Sl   Jan22   6:03
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  15535  0.0  0.4 39830952 427408 ?     Sl   Jan22   5:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins  20308  0.0  0.6 40012908 645164 ?     Sl   Jan22   5:29
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins  25889  0.0  0.9 40853360 977276 ?     Sl   Jan22   5:59
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  30415  0.0  0.8 41621188 869660 ?     Sl   Jan22   6:53
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  30575  0.0  0.6 40199036 636260 ?     Sl   Jan22   5:31
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins  31923  0.1  0.2 5073496 291664 ?      Sl   Jan26   0:45
> /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath
> /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf
> -Dmaven.home=/home/jenkins/tools/maven/latest3
> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14
> org.codehaus.plexus.classworlds.launcher.Launcher
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository
> -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999   32332  0.0  0.0  16788  2112 pts/0    S+   02:40   0:00 grep
> --color=auto CXF
>
>
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org>
>> wrote:
>>
>> Here is some data from H41, which was rebooted last night and ran out of
>> threads today. https://paste.apache.org/lkmpq
>>
>> In this case it looks like Karaf was still stuck/broken even though there
>> were no builds running on H41 at the time I investigated.
>>
>> -Chris T.
>> #asfinfra
>>
>>
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>>
>>>
>>> There are two Jenkins plugins that help with this situation
>>>  - CloudBees Disk Usage Simple
>>>  - Percentage Disk Space Node Column
>>>
>>> Could either/both of them be an option to install?
>>>
>>> regards,
>>> Mick
>>
>
>

Re: broken builds taking up resources

[Alexey Markevich <bu...@gmail.com>]

Hello,

I checked PR [1] and found many build failures on H41 during 'git'
command execution [2]. The last failure was OOM [3]. Finally build was
successful using H29 [4].

1. https://github.com/apache/cxf/pull/631
2. https://builds.apache.org/job/CXF-Trunk-PR/1389/console
3. https://builds.apache.org/job/CXF-Trunk-PR/1390/console
4. https://builds.apache.org/job/CXF-Trunk-PR/1391/

On 1/27/20, Chris Lambertus <cm...@apache.org> wrote:
> Here is some data from H24, which also contains many broken CXF jobs (not
> Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF
> builds, not karaf builds as previously noted. Copying dev@CXF since this
> build seems to be related to ongoing node problems.
>
>
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I
> have no idea what these are from, but they appear to be docker related.
>
>
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins   5951  0.0  0.8 40341652 813096 ?     Sl   Jan22   5:45
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins   6315  0.0  0.1 36400060 165960 ?     Sl   Jan22   4:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire
> 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp
> surefire_1029401344835483973390tmp
> jenkins   6326  0.0  0.6 41248112 634760 ?     Sl   Jan22   6:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins   7527  0.1  0.3 38255856 383992 ?     Sl   Jan22  10:52
> /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar
> /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire
> 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp
> surefire_359166059833874743459tmp
> jenkins   8669  0.0  0.6 40598416 635516 ?     Sl   Jan22   5:52
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  15485  0.0  0.6 41069240 646600 ?     Sl   Jan22   6:03
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  15535  0.0  0.4 39830952 427408 ?     Sl   Jan22   5:11
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins  20308  0.0  0.6 40012908 645164 ?     Sl   Jan22   5:29
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins  25889  0.0  0.9 40853360 977276 ?     Sl   Jan22   5:59
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  30415  0.0  0.8 41621188 869660 ?     Sl   Jan22   6:53
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar
> org.apache.karaf.main.Main
> jenkins  30575  0.0  0.6 40199036 636260 ?     Sl   Jan22   5:31
> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java
> -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed
> -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext
> -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances
> -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0
> -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data
> -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc
> -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp
> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar
> org.apache.karaf.main.Main
> jenkins  31923  0.1  0.2 5073496 291664 ?      Sl   Jan26   0:45
> /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath
> /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf
> -Dmaven.home=/home/jenkins/tools/maven/latest3
> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14
> org.codehaus.plexus.classworlds.launcher.Launcher
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository
> -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999   32332  0.0  0.0  16788  2112 pts/0    S+   02:40   0:00 grep
> --color=auto CXF
>
>
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org>
>> wrote:
>>
>> Here is some data from H41, which was rebooted last night and ran out of
>> threads today. https://paste.apache.org/lkmpq
>>
>> In this case it looks like Karaf was still stuck/broken even though there
>> were no builds running on H41 at the time I investigated.
>>
>> -Chris T.
>> #asfinfra
>>
>>
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>>
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>>
>>>
>>> There are two Jenkins plugins that help with this situation
>>>  - CloudBees Disk Usage Simple
>>>  - Percentage Disk Space Node Column
>>>
>>> Could either/both of them be an option to install?
>>>
>>> regards,
>>> Mick
>>
>
>

Re: broken builds taking up resources

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
> 
> 	This is almost always because whatever is running on the two executors have suffocated the system resources.

... and before I forget, a reminder:  Java threads take up a file descriptor. Hadoop's unit tests were firing up 10s of thousands of threads which were eating up 10s of thousands of FDs and ultimately lead to "cannot fork, no resource" errors causing everything to come tumbling down for the Jenkins salve process.  So _all_ the resources, not just RAM or whatever.

Re: broken builds taking up resources

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
> 
> 	This is almost always because whatever is running on the two executors have suffocated the system resources.

... and before I forget, a reminder:  Java threads take up a file descriptor. Hadoop's unit tests were firing up 10s of thousands of threads which were eating up 10s of thousands of FDs and ultimately lead to "cannot fork, no resource" errors causing everything to come tumbling down for the Jenkins salve process.  So _all_ the resources, not just RAM or whatever.

Re: broken builds taking up resources

[Chris Lambertus <cm...@apache.org>]

> On Jan 28, 2020, at 8:22 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:

[snip]

> [1] - The best on-prem solution I came up with  (before I moved my $DAYJOB stuff to cloud) was to run each executor in a VM on the box.  That VM would also have a regularly scheduled job that would cause it to wipe itself and respawn via a trigger mechanism.  Yeah, completely sucks, but at least it affords a lot more safety.

Allen,

I’d be interested in hearing more about your implementation here. Infra is moving towards a more compartmentalized strategy for jenkins, and I’m looking at how builds could be ephemeralized using k8s. 

-Chris

Re: broken builds taking up resources

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Jan 28, 2020, at 8:02 PM, Chris Lambertus <cm...@apache.org> wrote:
> 
> 
> Allen, can you elaborate on what a “proper” implementation is?  As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.
> 

	You are correct: the limitations come from the implementation of the jenkins slave jar. Ideally it would run the slave.jar as one user and executors as one or more users.  Or at least use cgroups on Linux and RBAC on Solaris and jails on FreeBSD and ... to at least do a minimal amount of work to protect itself.  Instead, it depends upon the good will of spawned processes to not shoot it or anything else running on the box.  This works great for the absolutely simple case, but completely false apart for anything beyond running a handful of shell commands.

	Thus why I consider it idiotic.  There are ways Jenkins could have done some work to prevent this situation from occurring, but alas that is not the case.  Yes, it would require more setup of the client, but for those places that need (i.e., most) it would have been worth it. 

	Instead, on-prem operators are pretty much forced to build a ton of complex machinery to prevent users from wreaking havoc. [1] Or give up and move to either Jenkins talking to cloud or dump Jenkins entirely.


[1] - The best on-prem solution I came up with  (before I moved my $DAYJOB stuff to cloud) was to run each executor in a VM on the box.  That VM would also have a regularly scheduled job that would cause it to wipe itself and respawn via a trigger mechanism.  Yeah, completely sucks, but at least it affords a lot more safety. 

Re: broken builds taking up resources

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Jan 28, 2020, at 8:02 PM, Chris Lambertus <cm...@apache.org> wrote:
> 
> 
> Allen, can you elaborate on what a “proper” implementation is?  As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.
> 

	You are correct: the limitations come from the implementation of the jenkins slave jar. Ideally it would run the slave.jar as one user and executors as one or more users.  Or at least use cgroups on Linux and RBAC on Solaris and jails on FreeBSD and ... to at least do a minimal amount of work to protect itself.  Instead, it depends upon the good will of spawned processes to not shoot it or anything else running on the box.  This works great for the absolutely simple case, but completely false apart for anything beyond running a handful of shell commands.

	Thus why I consider it idiotic.  There are ways Jenkins could have done some work to prevent this situation from occurring, but alas that is not the case.  Yes, it would require more setup of the client, but for those places that need (i.e., most) it would have been worth it. 

	Instead, on-prem operators are pretty much forced to build a ton of complex machinery to prevent users from wreaking havoc. [1] Or give up and move to either Jenkins talking to cloud or dump Jenkins entirely.


[1] - The best on-prem solution I came up with  (before I moved my $DAYJOB stuff to cloud) was to run each executor in a VM on the box.  That VM would also have a regularly scheduled job that would cause it to wipe itself and respawn via a trigger mechanism.  Yeah, completely sucks, but at least it affords a lot more safety. 

Re: broken builds taking up resources

[Chris Lambertus <cm...@apache.org>]

> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
> 
> 
> 
>> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
>> 
>> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
>> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
>> that could probably explain the hanging builds. 
> 
> 
> 	This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect.  (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed.  This is an idiotic implementation, but it is what it is.)

this is correct re: starving the slave.jar. It’s entirely likely the CXF jobs are the ones causing the disconnect based on available information. 

Allen, can you elaborate on what a “proper” implementation is?  As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.


-Chris

Re: broken builds taking up resources

[Chris Lambertus <cm...@apache.org>]

> On Jan 27, 2020, at 10:52 PM, Allen Wittenauer <aw...@effectivemachines.com.INVALID> wrote:
> 
> 
> 
>> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
>> 
>> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
>> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
>> that could probably explain the hanging builds. 
> 
> 
> 	This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect.  (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed.  This is an idiotic implementation, but it is what it is.)

this is correct re: starving the slave.jar. It’s entirely likely the CXF jobs are the ones causing the disconnect based on available information. 

Allen, can you elaborate on what a “proper” implementation is?  As far as I know, this is baked into jenkins. We could raise process limits for the jenkins user, but these situations only tend to arise when a build has gone off the rails.


-Chris

Re: broken builds taking up resources

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
> 
> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
> that could probably explain the hanging builds. 


	This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect.  (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed.  This is an idiotic implementation, but it is what it is.)

	Anyway, in my experiences, if all/most of one type of job are failing with  the node to appear to be crashed, then there is a good chance that job is the cause.  So it would be great if someone could spend the effort to profile the CXF jobs to see what their actual resource consumption is.

	FWIW, we had this problem with Hadoop, HBase, and others on the 'Hadoop' label nodes. The answer was to:

a) always run our jobs in containers that could be easily killed (freestyle Jenkins jobs that do 'docker run' generally can't be killed, despite what the UI says, because the signal never reaches the container)
b) those containers had resource limits 
c) increase the resources that systemd is allowed to give the jenkins user

	After doing that, the number of failures on the Hadoop nodes dropped exponentially. 

Re: broken builds taking up resources

[Allen Wittenauer <aw...@effectivemachines.com.INVALID>]

> On Jan 27, 2020, at 6:37 PM, Andriy Redko <dr...@gmail.com> wrote:
> 
> Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
> because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
> that could probably explain the hanging builds. 


	This is almost always because whatever is running on the two executors have suffocated the system resources. This ends up starving the Jenkins slave.jar, thus causing the disconnect.  (It's extremely important to understand that Jenkins' implementation here is sort of brain dead: the slave.jar runs as the SAME USER as the jobs being executed.  This is an idiotic implementation, but it is what it is.)

	Anyway, in my experiences, if all/most of one type of job are failing with  the node to appear to be crashed, then there is a good chance that job is the cause.  So it would be great if someone could spend the effort to profile the CXF jobs to see what their actual resource consumption is.

	FWIW, we had this problem with Hadoop, HBase, and others on the 'Hadoop' label nodes. The answer was to:

a) always run our jobs in containers that could be easily killed (freestyle Jenkins jobs that do 'docker run' generally can't be killed, despite what the UI says, because the signal never reaches the container)
b) those containers had resource limits 
c) increase the resources that systemd is allowed to give the jenkins user

	After doing that, the number of failures on the Hadoop nodes dropped exponentially. 

Re: broken builds taking up resources

[Andriy Redko <dr...@gmail.com>]

Hi Chris,

Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
that could probably explain the hanging builds. 

Best Regards,
    Andriy Redko

CL> Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed
CL> because of broken container jobs from 2019, with over 1100(!) processes identical to:

CL> containerd-shim -namespace moby -workdir
CL> /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc
CL> -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc

CL> /sigh



CL> asf999@asf940:~$ ps wwaxu | grep CXF
CL> jenkins   8380  0.0  0.5 39795216 541148 ?     Sl   Jan22   5:06
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
CL> jenkins  16834  0.0  0.5 39733804 496308 ?     Sl   Jan22   5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
CL> jenkins  25793  0.0  0.4 39797272 471988 ?     Sl   Jan22   5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
CL> jenkins  28293  0.0  0.5 39751220 521156 ?     Sl   Jan21   6:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
CL> jenkins  28431  0.1  1.2 6242916 1218032 ?     Sl   Jan23   6:35 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> jenkins  36663  0.0  0.5 39697712 556556 ?     Sl   Jan22   5:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
CL> jenkins  37958  0.0  0.5 4946924 567908 ?      Sl   Jan23   2:01 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> asf999   59270  0.0  0.0  16576  2088 pts/0    S+   05:39   0:00 grep --color=auto CXF



>> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:

>> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.


>> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.


>> asf999@asf924:~$ ps wwaxu | grep CXF
>> jenkins   5951  0.0  0.8 40341652 813096 ?     Sl   Jan22   5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins   6315  0.0  0.1 36400060 165960 ?     Sl   Jan22   4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
>> jenkins   6326  0.0  0.6 41248112 634760 ?     Sl   Jan22   6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins   7527  0.1  0.3 38255856 383992 ?     Sl   Jan22  10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
>> jenkins   8669  0.0  0.6 40598416 635516 ?     Sl   Jan22   5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  15485  0.0  0.6 41069240 646600 ?     Sl   Jan22   6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  15535  0.0  0.4 39830952 427408 ?     Sl   Jan22   5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins  20308  0.0  0.6 40012908 645164 ?     Sl   Jan22   5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins  25889  0.0  0.9 40853360 977276 ?     Sl   Jan22   5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  30415  0.0  0.8 41621188 869660 ?     Sl   Jan22   6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  30575  0.0  0.6 40199036 636260 ?     Sl   Jan22   5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins  31923  0.1  0.2 5073496 291664 ?      Sl   Jan26   0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
>> asf999   32332  0.0  0.0  16788  2112 pts/0    S+   02:40   0:00 grep --color=auto CXF


>>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:

>>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq

>>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.

>>> -Chris T.
>>> #asfinfra


>>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:

>>>>> Is there some way we can improve the visibility into disk usage on the
>>>>> build nodes? How full they are? And what projects are taking up space?
>>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>>> somewhere?


>>>> There are two Jenkins plugins that help with this situation
>>>> - CloudBees Disk Usage Simple
>>>> - Percentage Disk Space Node Column

>>>> Could either/both of them be an option to install?

>>>> regards,
>>>> Mick

Re: broken builds taking up resources

[Andriy Redko <dr...@gmail.com>]

Hi Chris,

Thanks a lot for looking into it. From the CXF perspective, I have seen that many CXF builds have been aborted
because of the connection with master is lost (don't have exact builds to point since we keep only last 3),
that could probably explain the hanging builds. 

Best Regards,
    Andriy Redko

CL> Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed
CL> because of broken container jobs from 2019, with over 1100(!) processes identical to:

CL> containerd-shim -namespace moby -workdir
CL> /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc
CL> -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc

CL> /sigh



CL> asf999@asf940:~$ ps wwaxu | grep CXF
CL> jenkins   8380  0.0  0.5 39795216 541148 ?     Sl   Jan22   5:06
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
CL> jenkins  16834  0.0  0.5 39733804 496308 ?     Sl   Jan22   5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
CL> jenkins  25793  0.0  0.4 39797272 471988 ?     Sl   Jan22   5:40
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
CL> jenkins  28293  0.0  0.5 39751220 521156 ?     Sl   Jan21   6:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
CL> jenkins  28431  0.1  1.2 6242916 1218032 ?     Sl   Jan23   6:35 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> jenkins  36663  0.0  0.5 39697712 556556 ?     Sl   Jan22   5:07
CL> /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar
CL> /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire
CL> 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
CL> jenkins  37958  0.0  0.5 4946924 567908 ?      Sl   Jan23   2:01 /home/jenkins/tools/java/latest11/bin/java
CL> -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar
CL> -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3
CL> -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native
CL> -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11
CL> org.codehaus.plexus.classworlds.launcher.Launcher
CL> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
CL> asf999   59270  0.0  0.0  16576  2088 pts/0    S+   05:39   0:00 grep --color=auto CXF



>> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:

>> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.


>> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.


>> asf999@asf924:~$ ps wwaxu | grep CXF
>> jenkins   5951  0.0  0.8 40341652 813096 ?     Sl   Jan22   5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins   6315  0.0  0.1 36400060 165960 ?     Sl   Jan22   4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
>> jenkins   6326  0.0  0.6 41248112 634760 ?     Sl   Jan22   6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins   7527  0.1  0.3 38255856 383992 ?     Sl   Jan22  10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
>> jenkins   8669  0.0  0.6 40598416 635516 ?     Sl   Jan22   5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  15485  0.0  0.6 41069240 646600 ?     Sl   Jan22   6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  15535  0.0  0.4 39830952 427408 ?     Sl   Jan22   5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins  20308  0.0  0.6 40012908 645164 ?     Sl   Jan22   5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins  25889  0.0  0.9 40853360 977276 ?     Sl   Jan22   5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  30415  0.0  0.8 41621188 869660 ?     Sl   Jan22   6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
>> jenkins  30575  0.0  0.6 40199036 636260 ?     Sl   Jan22   5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
>> jenkins  31923  0.1  0.2 5073496 291664 ?      Sl   Jan26   0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
>> asf999   32332  0.0  0.0  16788  2112 pts/0    S+   02:40   0:00 grep --color=auto CXF


>>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:

>>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq

>>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.

>>> -Chris T.
>>> #asfinfra


>>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:

>>>>> Is there some way we can improve the visibility into disk usage on the
>>>>> build nodes? How full they are? And what projects are taking up space?
>>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>>> somewhere?


>>>> There are two Jenkins plugins that help with this situation
>>>> - CloudBees Disk Usage Simple
>>>> - Percentage Disk Space Node Column

>>>> Could either/both of them be an option to install?

>>>> regards,
>>>> Mick

Re: broken builds taking up resources

[Chris Lambertus <cm...@apache.org>]

Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed because of broken container jobs from 2019, with over 1100(!) processes identical to:

containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc

/sigh



asf999@asf940:~$ ps wwaxu | grep CXF
jenkins   8380  0.0  0.5 39795216 541148 ?     Sl   Jan22   5:06 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
jenkins  16834  0.0  0.5 39733804 496308 ?     Sl   Jan22   5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
jenkins  25793  0.0  0.4 39797272 471988 ?     Sl   Jan22   5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
jenkins  28293  0.0  0.5 39751220 521156 ?     Sl   Jan21   6:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
jenkins  28431  0.1  1.2 6242916 1218032 ?     Sl   Jan23   6:35 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
jenkins  36663  0.0  0.5 39697712 556556 ?     Sl   Jan22   5:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
jenkins  37958  0.0  0.5 4946924 567908 ?      Sl   Jan23   2:01 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
asf999   59270  0.0  0.0  16576  2088 pts/0    S+   05:39   0:00 grep --color=auto CXF



> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:
> 
> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
> 
> 
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
> 
> 
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins   5951  0.0  0.8 40341652 813096 ?     Sl   Jan22   5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins   6315  0.0  0.1 36400060 165960 ?     Sl   Jan22   4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
> jenkins   6326  0.0  0.6 41248112 634760 ?     Sl   Jan22   6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins   7527  0.1  0.3 38255856 383992 ?     Sl   Jan22  10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
> jenkins   8669  0.0  0.6 40598416 635516 ?     Sl   Jan22   5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  15485  0.0  0.6 41069240 646600 ?     Sl   Jan22   6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  15535  0.0  0.4 39830952 427408 ?     Sl   Jan22   5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins  20308  0.0  0.6 40012908 645164 ?     Sl   Jan22   5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins  25889  0.0  0.9 40853360 977276 ?     Sl   Jan22   5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  30415  0.0  0.8 41621188 869660 ?     Sl   Jan22   6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  30575  0.0  0.6 40199036 636260 ?     Sl   Jan22   5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins  31923  0.1  0.2 5073496 291664 ?      Sl   Jan26   0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999   32332  0.0  0.0  16788  2112 pts/0    S+   02:40   0:00 grep --color=auto CXF
> 
> 
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>> 
>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>> 
>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>> 
>> -Chris T.
>> #asfinfra
>> 
>> 
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>> 
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>> 
>>> 
>>> There are two Jenkins plugins that help with this situation
>>> - CloudBees Disk Usage Simple
>>> - Percentage Disk Space Node Column
>>> 
>>> Could either/both of them be an option to install?
>>> 
>>> regards,
>>> Mick
>> 
> 

Re: broken builds taking up resources

[Chris Lambertus <cm...@apache.org>]

Another incident of CXF build junk sticking around on H40, although in that case, the machine appears to be hosed because of broken container jobs from 2019, with over 1100(!) processes identical to:

containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/a8bdf18973dbb865bb7d973faee38361af32372a186ad8618b0371fe26fd35cc -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc

/sigh



asf999@asf940:~$ ps wwaxu | grep CXF
jenkins   8380  0.0  0.5 39795216 541148 ?     Sl   Jan22   5:06 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2605123853901509690.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T02-17-36_679-jvmRun1 surefire2477874541156783106tmp surefire_355731680319470622184tmp
jenkins  16834  0.0  0.5 39733804 496308 ?     Sl   Jan22   5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2176472080093301198.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T13-57-24_910-jvmRun1 surefire1223758858590215789tmp surefire_352090757718230945819tmp
jenkins  25793  0.0  0.4 39797272 471988 ?     Sl   Jan22   5:40 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter2285236926740507372.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T01-17-55_297-jvmRun1 surefire6972444648312792386tmp surefire_351240114022660372795tmp
jenkins  28293  0.0  0.5 39751220 521156 ?     Sl   Jan21   6:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire/surefirebooter6319846263866409009.jar /home/jenkins/jenkins-slave/workspace/CXF-3.3.x-JDK18/rt/transports/http-netty/netty-server/target/surefire 2020-01-21T13-51-34_202-jvmRun1 surefire6933784412872895366tmp surefire_35221823887365033180tmp
jenkins  28431  0.1  1.2 6242916 1218032 ?     Sl   Jan23   6:35 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
jenkins  36663  0.0  0.5 39697712 556556 ?     Sl   Jan22   5:07 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire/surefirebooter4843455199269945966.jar /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T14-54-33_710-jvmRun1 surefire456414829921303412tmp surefire_34483670662819519029tmp
jenkins  37958  0.0  0.5 4946924 567908 ?      Sl   Jan23   2:01 /home/jenkins/tools/java/latest11/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK11/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
asf999   59270  0.0  0.0  16576  2088 pts/0    S+   05:39   0:00 grep --color=auto CXF



> On Jan 26, 2020, at 7:06 PM, Chris Lambertus <cm...@apache.org> wrote:
> 
> Here is some data from H24, which also contains many broken CXF jobs (not Karaf) from Jan 22. The builds on H41 use karaf artifacts, but they were CXF builds, not karaf builds as previously noted. Copying dev@CXF since this build seems to be related to ongoing node problems.
> 
> 
> Additionally, there are dozens of ‘unpigz’ processes stuck since 2019. I have no idea what these are from, but they appear to be docker related.
> 
> 
> asf999@asf924:~$ ps wwaxu | grep CXF
> jenkins   5951  0.0  0.8 40341652 813096 ?     Sl   Jan22   5:45 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/378d4d2a-e6ce-4ab1-9901-edee91067d74/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins   6315  0.0  0.1 36400060 165960 ?     Sl   Jan22   4:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire/surefirebooter4970188134198882072.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/systests/tracing/target/surefire 2020-01-22T01-21-20_250-jvmRun1 surefire13642588514701631413tmp surefire_1029401344835483973390tmp
> jenkins   6326  0.0  0.6 41248112 634760 ?     Sl   Jan22   6:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/7d7f3a1a-8ef5-4708-ac71-e24fa26af450/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins   7527  0.1  0.3 38255856 383992 ?     Sl   Jan22  10:52 /usr/local/asfpackages/java/openjdk-14-ea+17/bin/java -ea -Dnone= -jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire/surefirebooter11432107973602101312.jar /home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/rt/transports/http-netty/netty-server/target/surefire 2020-01-22T03-24-34_079-jvmRun1 surefire9072608189073279529tmp surefire_359166059833874743459tmp
> jenkins   8669  0.0  0.6 40598416 635516 ?     Sl   Jan22   5:52 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/ceb51bd8-c88d-45ee-b9bc-734d13dac4d7/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  15485  0.0  0.6 41069240 646600 ?     Sl   Jan22   6:03 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cfc415db-aead-46e0-9991-bdce39972f21/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  15535  0.0  0.4 39830952 427408 ?     Sl   Jan22   5:11 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/0ae722d3-b531-49cd-91d0-e20ea47f076d/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins  20308  0.0  0.6 40012908 645164 ?     Sl   Jan22   5:29 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/7b30e5fe-dfb0-4575-b606-e86054e4668a/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins  25889  0.0  0.9 40853360 977276 ?     Sl   Jan22   5:59 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/xkms/xkms-itests/target/paxexam/unpack/cf6d1769-7e25-495b-bdee-120bee163bba/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  30415  0.0  0.8 41621188 869660 ?     Sl   Jan22   6:53 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/etc/java.util.logging.properties -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=false -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.activator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/org.apache.servicemix.specs.locator-2.9.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-4.5.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/org.apache.karaf.main-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/services/sts/systests/sts-itests/target/paxexam/unpack/cccb4603-e977-4206-888e-eabaffcf0b1c/lib/boot/jna-platform-4.5.0.jar org.apache.karaf.main.Main
> jenkins  30575  0.0  0.6 40199036 636260 ?     Sl   Jan22   5:31 /usr/local/asfpackages/java/jdk1.8.0_191/jre/bin/java -Djava.endorsed.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/endorsed:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/endorsed:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/endorsed -Djava.ext.dirs=/usr/local/asfpackages/java/jdk1.8.0_191/jre/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext:/usr/local/asfpackages/java/jdk1.8.0_191/jre/lib/ext -Dkaraf.instances=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/instances -Dkaraf.home=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.base=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0 -Dkaraf.data=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/data -Dkaraf.etc=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc -Djava.util.logging.config.file=/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/etc/java.util.logging.properties -Dkaraf.startLocalConsole=true -Dkaraf.startRemoteShell=true -cp /home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.osgi.core-6.0.0.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.diagnostic.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.jaas.boot-4.1.7.jar:/home/jenkins/jenkins-slave/workspace/CXF-3.2.x/osgi/itests/target/paxexam/9c2f7326-226e-4e11-acb5-eae4b68dbdd0/lib/boot/org.apache.karaf.main-4.1.7.jar org.apache.karaf.main.Main
> jenkins  31923  0.1  0.2 5073496 291664 ?      Sl   Jan26   0:45 /home/jenkins/tools/java/latest14/bin/java -Xmx1024M -classpath /home/jenkins/tools/maven/latest3/boot/plexus-classworlds-2.6.0.jar -Dclassworlds.conf=/home/jenkins/tools/maven/latest3/bin/m2.conf -Dmaven.home=/home/jenkins/tools/maven/latest3 -Dlibrary.jansi.path=/home/jenkins/tools/maven/latest3/lib/jansi-native -Dmaven.multiModuleProjectDirectory=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14 org.codehaus.plexus.classworlds.launcher.Launcher -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/CXF-Master-JDK14/.repository -U -B clean install -Dmaven.test.skip.exec=true -Pnochecks
> asf999   32332  0.0  0.0  16788  2112 pts/0    S+   02:40   0:00 grep --color=auto CXF
> 
> 
>> On Jan 24, 2020, at 1:57 PM, Chris Thistlethwaite <ch...@apache.org> wrote:
>> 
>> Here is some data from H41, which was rebooted last night and ran out of threads today. https://paste.apache.org/lkmpq
>> 
>> In this case it looks like Karaf was still stuck/broken even though there were no builds running on H41 at the time I investigated.
>> 
>> -Chris T.
>> #asfinfra
>> 
>> 
>> On 1/24/20 4:26 AM, Mick Semb Wever wrote:
>>> 
>>>> Is there some way we can improve the visibility into disk usage on the
>>>> build nodes? How full they are? And what projects are taking up space?
>>>> Does jenkins provide this info? Or could infra dump a `du …` report
>>>> somewhere?
>>> 
>>> 
>>> There are two Jenkins plugins that help with this situation
>>> - CloudBees Disk Usage Simple
>>> - Percentage Disk Space Node Column
>>> 
>>> Could either/both of them be an option to install?
>>> 
>>> regards,
>>> Mick
>> 
>