You are viewing a plain text version of this content. The canonical link for it is here.

Posted to builds@apache.org by Aled Sage <al...@cloudsoftcorp.com> on 2018/09/24 14:20:16 UTC

Re: Empire-db build problems <-- brooklyn-master-build-docker_ws-cleanup workspace left behind

Hi Chris, all,

Thomas has just pointed me at 
https://issues.apache.org/jira/browse/INFRA-16417, in which you recommend:

    "The problem is the bind mount for .m2 and permissions conflicts
    between the docker
    image and jenkins. Please don't use a bind mount for filesystems you
    intend to write to.
    It will make the build run a bit longer since it has to download the
    artifacts within the
    container, but it will prevent you from having problems and
    conflicts with the
    jenkins .m2 directory."

Apologies for us not running with this advice previously - I'm looking 
into how we could do this.

Are there any good example configurations of other Apache projects 
building and running tests (ideally using maven) in docker containers, 
on apache jenkins slaves?

Aled


On 21/09/2018 10:03, Aled Sage wrote:
> Hi all,
>
> Do any brooklyn folk know what the 
> `brooklyn-master-build-docker_ws-cleanup_.*` workspaces are for, and 
> why they are not being cleaned up?
>
> It sounds like this bug in ws-cleanup-plugin [1]. It was fixed, but 
> then folk reported that they were still experiencing the same problem 
> [2].
>
> A workaround described in [3] is:
>
>   ws-cleanup is leaking workspaces because the dynamic nodes are 
> unavailable at the time
>   the dispose is ran. I've installed a cron which is run every 1/2/h 
> which is removing
>   ws-cleanup folders that aren't modified the last hour on ovh docker 
> hosts.
>
> I'm not sure what we could do on the Brooklyn side.
>
> ---
> Looking in the console output of our brooklyn-master-build-docker 
> jobs, I see #700 [4] says at the very start:
>
>   Building remotely on H24 (ubuntu xenial) in workspace 
> /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker
>   [WS-CLEANUP] Deleting project workspace...
>   [WS-CLEANUP] Done
>
> In contrast, builds #702 [5] doesn't have the line `[WS-CLEANUP] 
> Done`. Does that mean it will have left behind one of these 
> directories perhaps?
>
> Looking at [6], it looks like we only keep a history of 5 builds in 
> jenkins. We therefore should really not have 28 workspace directories 
> on the server H24!
>
> If it might be a problem with ws-cleanup having permissions to delete 
> the workspace, could it be related to the way we mount the workspace 
> as a volume in the docker container?
>
>   docker run -i --rm -v /home/jenkins:/usr/home
>       -v 
> /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker/.m2:/root/.m2 
> \
>       -v 
> /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker:/usr/build 
> \
> -w /usr/build brooklyn:latest mvn clean install -s 
> /usr/home/.m2/settings.xml
>
> Aled
>
> [1] https://issues.jenkins-ci.org/browse/JENKINS-24824
> [2] https://issues.jenkins-ci.org/browse/JENKINS-53579
> [3] https://jira.nuxeo.com/browse/NXBT-2205
> [4] 
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
> [5] 
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
> [6] https://builds.apache.org/job/brooklyn-master-build-docker/
>
>
> On 20/09/2018 20:20, Chris Lambertus wrote:
>>
>>
>>
>>
>>> On Sep 20, 2018, at 11:51 AM, Dave Brondsema <da...@brondsema.net> 
>>> wrote:
>>>
>>> Moving to builds@
>>>
>>> I'm seeing this type of disk error on Allura builds that have used 
>>> H27 and H24
>>> today.
>> Here’s the raw usage output from H24. Not sure what’s going on with 
>> those Brooklyn jobs. CC to dev@brooklyn so they can have a look.
>>
>>
>>
>> root@asf924:/home/jenkins/jenkins-slave/workspace# du -sh -- *
>> 4.3G    ActiveMQ-Java8
>> 2.5G    ActiveMQ-Java8-All-UnitTests
>> 1.2G    ActiveMQ-PR-Build
>> 4.5G    Ambari-branch-2.7
>> 8.0G    Ambari-trunk-Commit
>> 127M    Ant-Build-Matrix-1.9.x-Linux
>> 129M    Ant-Build-Matrix-master-Linux
>> 88M    Ant-Master SonarQube
>> 4.7M    Apache%20Jackrabbit%20Oak%20matrix
>> 15G    Apache Jackrabbit Oak matrix
>> 12M    Aries jax-rs whiteboard
>> 14M    Aries-rsa master deploy
>> 816M    aurora-packaging-nightly
>> 4.0K 
>> aven-assembly-plugin_master-SP4DXUL4GVN3Q5KJHI5ED6WPWUEC2L2PUW2RCT2SFVGTCXHUIFDQ@tmp
>> 4.0K 
>> aven-box_maven-doxia_master-4SHBBM33BQC2SGN65AA3AW6GPXIK6RA462YQND5AN3YPZ5U2VGSQ@tmp
>> 4.0K 
>> aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@2@tmp
>> 4.0K 
>> aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@tmp
>> 4.0K 
>> aven-enforcer_MENFORCER-312-H6BWWMKU3O4RIVLVOSDDUVD7CT73H5PSEGRJICFTH5FH5BR7L3XQ@tmp
>> 122M    Bigtop-trunk-testartifacts
>> 924M    bookkeeper_postcommit_master_java8
>> 261M    bookkeeper_precommit_bookie_tests
>> 302M    bookkeeper_precommit_client_tests
>> 754M    bookkeeper_precommit_pullrequest_java8
>> 758M    bookkeeper_precommit_pullrequest_java9
>> 272M    bookkeeper_precommit_pullrequest_validation
>> 400M    bookkeeper_precommit_pullrequest_validation@2
>> 334M    bookkeeper_precommit_remaining_tests
>> 257M    bookkeeper_precommit_replication_tests
>> 259M    bookkeeper_precommit_tls_tests
>> 907M    bookkeeper_release_branch_48_integrationtests
>> 886M    bookkeeper_release_branch_48_java8
>> 4.0K 
>> box_maven-ant-plugin_master-N743QKOWXTZ36LZCK2YK3A6N4ZY3UG7X5RV3MH7PEH646A2IMDHA@tmp
>> 4.0K 
>> -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@2@tmp
>> 4.0K 
>> -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@tmp
>> 4.0K 
>> box_maven-ejb-plugin_master-D5MGOBWWYZCBR23TA7FG7MUJQK233O27B5TSPA7YUOAEWLP5Z72Q@tmp
>> 4.0K 
>> -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@2@tmp
>> 4.0K 
>> -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@tmp
>> 5.7M 
>> -box_maven_pre-reset-master-AF72GWR2EBH4F7AYOLQPRDHKOEZISK7V6BEF33K5DPJKW2VE4ATA
>> 4.0K 
>> box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@2@tmp
>> 4.0K 
>> box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@tmp
>> 4.0K 
>> -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@2@tmp
>> 4.0K 
>> -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@tmp
>> 4.0K 
>> box_maven-war-plugin_master-QSYAUU4F27NQH3VKQKYR34QIY6P36LH6ACXOADA2HDWCZVVU4TYA@tmp
>> 155M    brooklyn-library-master-docker
>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535791870056
>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535929381454
>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535991522759
>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536015781206
>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536050912838
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536188581290
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536350369118
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536361380993
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536447781251
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536534181007
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536620581304
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536706981637
>> 451M    brooklyn-master-build-docker_ws-cleanup_1536793381496
>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536966181681
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1536974386104
>> 733M    brooklyn-master-build-docker_ws-cleanup_1537055864223
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537079895664
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537087960702
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537092848128
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537116168718
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537138981521
>> 3.1G    brooklyn-master-build-docker_ws-cleanup_1537166290489
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537252717591
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537260673390
>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537289677014
>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537425657042
>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537433463888
>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537461822531
>> 1.4G    Calcite-Master
>> 170M 
>> _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q
>> 8.0K 
>> _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q@tmp
>> 2.8G Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA
>> 4.0K 
>> Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA@tmp
>> 171M 
>> Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ
>> 4.0K 
>> Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ@tmp
>> 60M 
>> Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A
>> 4.0K 
>> Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A@tmp
>> 1.9G 
>> Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ
>> 4.0K 
>> Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ@tmp
>> 2.5G 
>> Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ
>> 4.0K 
>> Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ@tmp
>> 3.3G    Camel.daily
>> 4.0K    Camel.daily@tmp
>> 2.4G Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q
>> 4.0K 
>> Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q@tmp
>> 2.3G Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ
>> 4.0K 
>> Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ@tmp
>> 810M    carbondata-master-spark-2.1
>> 4.0K    carbondata-master-spark-2.1@tmp
>> 584K    commons-geometry
>> 4.0K 
>> CouchDB_master-FP4LRZTK2IWL4WCWS25JHGLJVPORIFX6ES6POVJARPVUTCZMU7SA@tmp
>> 190M    CXF-Master-Analysis
>> 188M    CXF-Master-JDK10
>> 24M    CXF-Master-JDK9
>> 1.7G    CXF-trunk-deploy
>> 2.2G    CXF-Trunk-JDK18
>> 74M    DeltaSpike Deploy
>> 22M    DeltaSpike for Java EE
>> 53M    DeltaSpike OWB 1.7.2
>> 70M    DeltaSpike OWB 2.0.6
>> 4.0K 
>> -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@2@tmp
>> 4.0K 
>> -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@tmp
>> 7.1M    Derby-JaCoCo
>> 7.3M    dir-ldap-api-jdk8-linux SONAR
>> 23M    dir-ldap-api-ubuntu-deploy
>> 2.0G    dir-studio
>> 2.7G    dir-studio-ui-tests-in-docker
>> 1.3M    Empire-db reports
>> 19M    Empire-db snapshot
>> 4.0K 
>> en-artifact-resolver_master-TCRLMCKTTCPXV2IPJG6T7K54FQQY2O7AFHHEFILBSO6ZFSIUCGRQ@tmp
>> 4.0K 
>> en-box_maven-mapping_master-VTBNHOZQB7UZ4ORPSWLQNN55JVWPXJBJNR54WLBERZKMDQGEPMXA@tmp
>> 4.0K 
>> en-jxr_asfMavenTlpPlgnBuild-WCSOEZUCROETKKJCNPTYSFNDV52TA7MIFL63KMUYVDEPFRT5OK4Q@tmp
>> 4.0K 
>> en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@2@tmp
>> 4.0K 
>> en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@tmp
>> 4.0K 
>> en-toolchains-plugin_master-JSJRH7N7WHA5OITGTMSQVCI6AVNUPHTOLBYXKJXWNGGNGCYHSF5A@tmp
>> 287M 
>> g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA
>> 4.0K 
>> g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA@tmp
>> 419M    guacamole-rest-api-test
>> 997M 
>> HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ
>> 4.0K 
>> HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ@tmp
>> 333M 
>> HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A
>> 4.0K 
>> HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A@tmp
>> 2.6G 
>> HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q
>> 8.0K 
>> HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q@tmp
>> 36M 
>> HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ
>> 4.0K 
>> HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ@tmp
>> 1.4G 
>> HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q
>> 4.0K 
>> HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q@tmp
>> 596K 
>> HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ
>> 4.0K 
>> HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ@tmp
>> 2.8G 
>> HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA
>> 4.0K 
>> HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@tmp
>> 538M    Hive-trunk
>> 3.2G    incubator-netbeans-linux
>> 744M    incubator-netbeans-linux-experiment
>> 4.0K 
>> -info-reports-plugin_master-32Y3P743XLZN3V3HHV4NR6ZSZMFDM7Z5PXO3RSN6SA3U6NVU4ZTA@tmp
>> 108K    infra-test-ant-ubuntu
>> 12K    infra-test-docker-ubuntu
>> 60K    infra-test-jdk-ubuntu
>> 20K    infra-test-jdk-ubuntu-harmony
>> 40K    infra-test-jdk-ubuntu-ibm
>> 56K    infra-test-maven-ubuntu
>> 12K    infra-test-snap-ubuntu
>> 12K    infra-test-xvfb
>> 1.4M    Ivy-Snapshot-Deploy
>> 99M    Ivy-tests-ubuntu
>> 16M    Ivy-tests-Windows
>> 12M    Jackrabbit-trunk
>> 4.0K 
>> javadoc-plugin_MJAVADOC-444-JPYTCQM73CF7UEOSYS6Z2MLTMHWVHYCB5QGVIGAH4MONP6SX7OHA@tmp
>> 4.0K 
>> javadoc-plugin_MJAVADOC-516-Z27AS7O3YE3KP4HN2X3EMP56QPDDJR4P5EBKM2B5MKLXDNLXZ3DA@tmp
>> 19M    Jena_Development_Test
>> 731M    Jena_JDK11
>> 264M    kafka-1.0-jdk7
>> 290M    kafka-1.1-jdk7
>> 501M    kafka-2.0-jdk8
>> 628M    kafka-pr-jdk10-scala2.12
>> 582M    kafka-pr-jdk8-scala2.11
>> 523M    kafka-trunk-jdk8
>> 60M    karaf-cave-master
>> 4.8G    karaf-master
>> 344M    Log4j 2 3.x
>> 173M 
>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ
>> 93M 
>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2
>> 4.0K 
>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2@tmp
>> 4.0K 
>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@tmp
>> 4.0K 
>> ly-plugin_FIXING-JDK-ISSUES-IEX46HBDBJ6NRUCALBPK7SVRYXYL7WLKQAOOVWNB63FAXVHXQWMQ@tmp
>> 11M 
>> maticModuleNameImprovements-5CTZJI7MCAG23DNAIMMOTYFEI4EKBOZ2AOP3EOG34Q46K3EBQVBA
>> 4.0K 
>> _maven-archiver_MSHARED-721-QXH4Z3WHL2IJTFJJIFNELXYZM6YOEOOXEJCOQA7JK5JMIS42PNSA@tmp
>> 4.0K 
>> maven-box_maven-jxr_JXR-110-R2OIXR7IG7PF3QUMPP2IV53ZAKEOGWXENXIP7G2LROL7YOEOZZTA@tmp
>> 4.0K 
>> maven-box_maven-jxr_JXR-125-KFC5JZL777UAPG4UY6F434EV6BJ2ZRPG7HBO32YUS3UJ56NWROZQ@tmp
>> 4.0K 
>> maven-box_maven-jxr_master-HV7BHOKN2YNGQ4SOGMW6QYUNHKCZDFS3I74XW3IMRQXBAUKY6MSQ@tmp
>> 8.0K 
>> maven-box_maven_master-DVEN5KAF4FLE3WDK7SWFNYUY4YZ7X45JVWYY6FZT5ZEB3CLWABVQ
>> 5.7M 
>> maven-box_maven_MNG-5868-TIXVUCFHOPHJ7LSBIS4NVVVTXHOVMMQO4W2BXVF455AQYGYZ26SA
>> 8.0K 
>> maven-box_maven_MNG-6391-BFSVDMPDHLGZMTGMVMMLWPKG4K2P6CHHPIPVW27SUVBB6UQWV7YA
>> 8.0K 
>> maven-box_maven_MNG-6475-LMWI4CBOH2JNDHX4PPHOWPHEASG5JKLTTZW3URJSHEBTPFA2447Q
>> 4.0K 
>> maven-box_maven-osgi_master-537V4ALFOX2ZNQAIHHNQL5TC7LIKAHUUTBK4WE42PW2CUAN3GTBQ@tmp
>> 4.0K 
>> maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@2@tmp
>> 4.0K 
>> maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@tmp
>> 4.0K 
>> maven-changes-plugin_master-O3PQ7WZZHAA2T5PBIGGQZFFXFD6PLMGLZHAF4U525J6NJFJA5IMQ@tmp
>> 4.0K 
>> maven-install-plugin_master-3R4FSR6ITADJX4LXJM433BEETYXHTGVGRU6VEEEHH2PU3W5LNT6A@tmp
>> 4.0K 
>> _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@2@tmp
>> 4.0K 
>> _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@tmp
>> 4.0K 
>> _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@2@tmp
>> 4.0K 
>> _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@tmp
>> 4.0K 
>> maven-reporting-exec_master-XY3GEBFQ42VNXZOS5MVD7PZKI6HQ7WOMNLQJ5IUZ7XSHVEI5U6RA@tmp
>> 5.0G    Mesos-Buildbot
>> 1.2M    Mesos-Reviewbot
>> 4.0K 
>> mpiler-plugin_MCOMPILER-320-LDDLDH7HQQYXD6B2RHESP35HSEBJ26NA355LSJP2PHGG3DW43OPQ@tmp
>> 4.0K 
>> m-publish-plugin_MSCMPUB-38-ULK6I7N2ZA3FU5JGEWCSL4ODBTZDTTC77NU7CUHQJKPNE7QVYS7A@tmp
>> 4.0K 
>> n-archiver_MSHARED-668-test-WG4DQP6EJS2N73J3VGM2JH6SMRRHF477RDRLQRWF4WAHIMWMG5OA@tmp
>> 4.0K 
>> n-box_maven-enforcer_master-KHI64AG3D4VZKUKLMNAYHKONI5AIQBLJ5ZKYGGSW7DAZVXCJBFHQ@tmp
>> 4.0K 
>> n-box_maven-release_cleanup-YGG2KC5W5NBVLT77VULX2CHSYMVGRKDIZ7FJM54R7TMFDTDO6JYA@tmp
>> 4.0K 
>> n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@2@tmp
>> 4.0K 
>> n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@tmp
>> 4.0K 
>> n-box_maven-wagon_WAGON-482-U5UQDLHLBEYZ6YKA2XFN2FQ74OK76EBRADZ4AK62MZWPJOQW2HDQ@tmp
>> 4.0K 
>> n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@2@tmp
>> 4.0K 
>> n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@tmp
>> 4.0K 
>> n-box_maven-wagon_WAGON-527-KWYDRBGRPK7Q2YVCYHYJDNDOSEOHGQPJHV5AHJB4GNG66RKBHCBA@tmp
>> 28K    Nemo
>> 4.0K 
>> nfo-reports-plugin_MPIR-366-S464FI4ATSVOULE3N7MCKV5FYYG6GVHDD7OLISMYBGUEPLQM2UNA@tmp
>> 5.7M 
>> _not-updated-MJAR-MCOMPILER-EYMH2UKFHCTFDWW4GZGG3AX4OJLGLPGZO6TAQGLBDBFX4K6CQ5GQ
>> 4.0K 
>> n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@2@tmp
>> 4.0K 
>> n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@tmp
>> 4.0K 
>> n-scm-publish-plugin_master-2GNHI3CFRHSCHWWMDIRTSWMJIE3KK6DFWPSDNLSGKYY2ZVCACIZA@tmp
>> 4.0K 
>> n-testing_MPLUGINTESTING-55-AWMCSLLBQGJRKNRZR27EZX2B5SVGADVMVK4FGA6YEL5S4OR3TOZQ@tmp
>> 4.0K 
>> n-testing_MPLUGINTESTING-58-UONNAARVYXAHZ4RVLUBM5ODIM6A4TFB5CEC4ELRKMGU3KDYXAPIQ@tmp
>> 800K 
>> nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA
>> 4.0K 
>> nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA@tmp
>> 343M    OpenWhisk-DockerHub
>> 4.0K    OpenWhisk-DockerHub@tmp
>> 4.0K 
>> ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@2@tmp
>> 4.0K 
>> ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@tmp
>> 4.0K 
>> ox_maven-fluido-skin_master-6RM5JATOOS6MTECVSVHV7YNCQIINRWOBDST2UCUIKNH64TYTME6Q@tmp
>> 4.0K 
>> ox_maven-help-plugin_master-U66QJWH3TRYC4M2J55DEV6AO7VQ2JW77H5GHKBUOBKZTW32WDTTA@tmp
>> 4.0K 
>> ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@2@tmp
>> 4.0K 
>> ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@tmp
>> 4.0K 
>> ox_maven-site-plugin_master-LZWAKOPRMO5JAZW5HRA2IQ56LLTFU3PLKDIE2JNFFXEGASBYKK3Q@tmp
>> 667M    PDFBox-trunk
>> 171M    PDFBox-Trunk-jdk10
>> 52M    Phoenix-4.x-HBase-1.2
>> 1.6G    Phoenix-4.x-HBase-1.3
>> 6.7M    PreCommit-Admin
>> 2.4G pulsar_precommit_integrationtests_ws-cleanup_1537313772888
>> 4.4G pulsar_precommit_integrationtests_ws-cleanup_1537469839992
>> 331M    Qpid-Broker-J-7.0.x-TestMatrix
>> 331M    Qpid-Broker-J-Artefact-Release
>> 339M    Qpid-Broker-J-JoramJMSTest
>> 733M    Qpid-Broker-J-Python-Test
>> 240M    Qpid-Broker-J-Test-IBMJDK1.8
>> 12M    Qpid-Java-Java-6.1.x-TestMatrix
>> 29M    Qpid-proton-c
>> 1.3M    Qpid-proton-j
>> 12M    ranger-0.7-snapshot-publish
>> 256M    Ratis-Master
>> 4.0K 
>> rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@2@tmp
>> 4.0K 
>> rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@tmp
>> 125M    samza-pr-checks
>> 4.0K 
>> sembly-plugin_MASSEMBLY-892-AIWNXXLMVTBQ62GDGAZ5RR52AYYD2PS4MNOQBOW2K6A6AS4WHTGQ@tmp
>> 50M    ServiceMix-Bundles
>> 36M 
>> sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A
>> 4.0K 
>> sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A@tmp
>> 282M    Struts-2-3-JDK6
>> 4.0K 
>> -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@2@tmp
>> 4.0K 
>> -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@tmp
>> 16M    Syncope-2_0_X-deploy
>> 4.3G    Syncope-2_1_X-deploy
>> 16M    Syncope-master-deploy
>> 21M    Tamaya-Javadoc-Master
>> 23M    Tamaya-Master-branch
>> 74M    Tobago Trunk
>> 81M    Tobago Trunk Sonar
>> 4.0K    Tomcat-7.x-Maven
>> 350M    trafficcontrol-master-build
>> 4.0K    trafficcontrol-master-build@tmp
>> 115M    trafficcontrol-PR-rat
>> 41M    trafficcontrol-PR-trafficops-test
>> 162M    trafficcontrol-traffic_ops-test
>> 4.0K    trafficcontrol-traffic_ops-test@tmp
>> 590M    unomi-master
>> 4.0K 
>> ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@2@tmp
>> 4.0K 
>> ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@tmp
>> 4.0K 
>> ven-box_maven-surefire_1532-O6EC5ODCIKDKZL25QDLH6QFMFZD5LDHASAYEMDZSNREZV4N37X7A@tmp
>> 4.0K 
>> ven-scripting-plugin_master-F7DFWWXZD7KAQGDPAQML3SYNOE2CUDOJDQGTJN7NKKPXY3SWUHCQ@tmp
>> 4.0K 
>> ven-site-plugin_MNGSITE-332-4LYPVCGK77QO5G6UKI6IR7JSEN5EMK7EVKG6TK2D3VYBAHH6QPBA@tmp
>> 4.0K 
>> x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@2@tmp
>> 4.0K 
>> x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@tmp
>> 4.0K 
>> x_maven-invoker_MSHARED-752-O2LI4W4PL6ARSMCGVOOVX6GZJCOYVGXMTNU5KTGD65M6CNMTDSRQ@tmp
>> 4.0K 
>> x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@2@tmp
>> 4.0K 
>> x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@tmp
>> 4.0K 
>> x_maven-jlink-plugin_master-TWLBBVLGC7P5GUJSJEP6SD46MW2WP64SO5MED7OJBUOM66RIHIJQ@tmp
>> 4.0K 
>> x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@2@tmp
>> 4.0K 
>> x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@tmp
>> 4.0K 
>> x_maven-release_IMPROVE_POM-KZP6VA3ZSGGYNTVVOCGLUBHYECJZQ34VIS6H2U5HNYVKLJFCFQKQ@tmp
>> 4.0K 
>> x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@2@tmp
>> 4.0K 
>> x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@tmp
>> 4.0K 
>> x_maven-stage-plugin_master-DUIB3VWMFIP4K7X4DPTLYSUTRVJKIKKQPIMXJ3UK6V5VUGFJHXHQ@tmp
>> 4.0K 
>> x_maven-wagon_wagon-scm-git-JFLMMYBLUJZHZUM4KST663ZOB5W6QQB2OF5JLK4YVV3OMUVRSJDA@tmp
>> 340M    Yetus - OWASP dependency-check cache
>> 119M    zeppelin-pull-request
>> 93M    ZooKeeper-Find-Flaky-Tests
>> 465M    ZooKeeper-trunk-openjdk10
>> 157M    ZooKeeper-trunk-owasp
>> 119M    ZooKeeper_UT_Stress
>

Re: Empire-db build problems <-- brooklyn-master-build-docker_ws-cleanup workspace left behind

Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.

This is pretty much what Apache Yetus was built to do. (I’d go into more depth but about to board a plane. ) :D

Sent from my ☎️

> On Sep 24, 2018, at 3:20 PM, Aled Sage <al...@cloudsoftcorp.com> wrote:
> 
> Hi Chris, all,
> 
> Thomas has just pointed me at https://issues.apache.org/jira/browse/INFRA-16417, in which you recommend:
> 
>   "The problem is the bind mount for .m2 and permissions conflicts
>   between the docker
>   image and jenkins. Please don't use a bind mount for filesystems you
>   intend to write to.
>   It will make the build run a bit longer since it has to download the
>   artifacts within the
>   container, but it will prevent you from having problems and
>   conflicts with the
>   jenkins .m2 directory."
> 
> Apologies for us not running with this advice previously - I'm looking into how we could do this.
> 
> Are there any good example configurations of other Apache projects building and running tests (ideally using maven) in docker containers, on apache jenkins slaves?
> 
> Aled
> 
> 
>> On 21/09/2018 10:03, Aled Sage wrote:
>> Hi all,
>> 
>> Do any brooklyn folk know what the `brooklyn-master-build-docker_ws-cleanup_.*` workspaces are for, and why they are not being cleaned up?
>> 
>> It sounds like this bug in ws-cleanup-plugin [1]. It was fixed, but then folk reported that they were still experiencing the same problem [2].
>> 
>> A workaround described in [3] is:
>> 
>>   ws-cleanup is leaking workspaces because the dynamic nodes are unavailable at the time
>>   the dispose is ran. I've installed a cron which is run every 1/2/h which is removing
>>   ws-cleanup folders that aren't modified the last hour on ovh docker hosts.
>> 
>> I'm not sure what we could do on the Brooklyn side.
>> 
>> ---
>> Looking in the console output of our brooklyn-master-build-docker jobs, I see #700 [4] says at the very start:
>> 
>>   Building remotely on H24 (ubuntu xenial) in workspace /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker
>>   [WS-CLEANUP] Deleting project workspace...
>>   [WS-CLEANUP] Done
>> 
>> In contrast, builds #702 [5] doesn't have the line `[WS-CLEANUP] Done`. Does that mean it will have left behind one of these directories perhaps?
>> 
>> Looking at [6], it looks like we only keep a history of 5 builds in jenkins. We therefore should really not have 28 workspace directories on the server H24!
>> 
>> If it might be a problem with ws-cleanup having permissions to delete the workspace, could it be related to the way we mount the workspace as a volume in the docker container?
>> 
>>   docker run -i --rm -v /home/jenkins:/usr/home
>>       -v /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker/.m2:/root/.m2 \
>>       -v /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker:/usr/build \
>> -w /usr/build brooklyn:latest mvn clean install -s /usr/home/.m2/settings.xml
>> 
>> Aled
>> 
>> [1] https://issues.jenkins-ci.org/browse/JENKINS-24824
>> [2] https://issues.jenkins-ci.org/browse/JENKINS-53579
>> [3] https://jira.nuxeo.com/browse/NXBT-2205
>> [4] https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
>> [5] https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
>> [6] https://builds.apache.org/job/brooklyn-master-build-docker/
>> 
>> 
>>> On 20/09/2018 20:20, Chris Lambertus wrote:
>>> 
>>> 
>>> 
>>> 
>>>> On Sep 20, 2018, at 11:51 AM, Dave Brondsema <da...@brondsema.net> wrote:
>>>> 
>>>> Moving to builds@
>>>> 
>>>> I'm seeing this type of disk error on Allura builds that have used H27 and H24
>>>> today.
>>> Here’s the raw usage output from H24. Not sure what’s going on with those Brooklyn jobs. CC to dev@brooklyn so they can have a look.
>>> 
>>> 
>>> 
>>> root@asf924:/home/jenkins/jenkins-slave/workspace# du -sh -- *
>>> 4.3G    ActiveMQ-Java8
>>> 2.5G    ActiveMQ-Java8-All-UnitTests
>>> 1.2G    ActiveMQ-PR-Build
>>> 4.5G    Ambari-branch-2.7
>>> 8.0G    Ambari-trunk-Commit
>>> 127M    Ant-Build-Matrix-1.9.x-Linux
>>> 129M    Ant-Build-Matrix-master-Linux
>>> 88M    Ant-Master SonarQube
>>> 4.7M    Apache%20Jackrabbit%20Oak%20matrix
>>> 15G    Apache Jackrabbit Oak matrix
>>> 12M    Aries jax-rs whiteboard
>>> 14M    Aries-rsa master deploy
>>> 816M    aurora-packaging-nightly
>>> 4.0K aven-assembly-plugin_master-SP4DXUL4GVN3Q5KJHI5ED6WPWUEC2L2PUW2RCT2SFVGTCXHUIFDQ@tmp
>>> 4.0K aven-box_maven-doxia_master-4SHBBM33BQC2SGN65AA3AW6GPXIK6RA462YQND5AN3YPZ5U2VGSQ@tmp
>>> 4.0K aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@2@tmp
>>> 4.0K aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@tmp
>>> 4.0K aven-enforcer_MENFORCER-312-H6BWWMKU3O4RIVLVOSDDUVD7CT73H5PSEGRJICFTH5FH5BR7L3XQ@tmp
>>> 122M    Bigtop-trunk-testartifacts
>>> 924M    bookkeeper_postcommit_master_java8
>>> 261M    bookkeeper_precommit_bookie_tests
>>> 302M    bookkeeper_precommit_client_tests
>>> 754M    bookkeeper_precommit_pullrequest_java8
>>> 758M    bookkeeper_precommit_pullrequest_java9
>>> 272M    bookkeeper_precommit_pullrequest_validation
>>> 400M    bookkeeper_precommit_pullrequest_validation@2
>>> 334M    bookkeeper_precommit_remaining_tests
>>> 257M    bookkeeper_precommit_replication_tests
>>> 259M    bookkeeper_precommit_tls_tests
>>> 907M    bookkeeper_release_branch_48_integrationtests
>>> 886M    bookkeeper_release_branch_48_java8
>>> 4.0K box_maven-ant-plugin_master-N743QKOWXTZ36LZCK2YK3A6N4ZY3UG7X5RV3MH7PEH646A2IMDHA@tmp
>>> 4.0K -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@2@tmp
>>> 4.0K -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@tmp
>>> 4.0K box_maven-ejb-plugin_master-D5MGOBWWYZCBR23TA7FG7MUJQK233O27B5TSPA7YUOAEWLP5Z72Q@tmp
>>> 4.0K -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@2@tmp
>>> 4.0K -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@tmp
>>> 5.7M -box_maven_pre-reset-master-AF72GWR2EBH4F7AYOLQPRDHKOEZISK7V6BEF33K5DPJKW2VE4ATA
>>> 4.0K box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@2@tmp
>>> 4.0K box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@tmp
>>> 4.0K -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@2@tmp
>>> 4.0K -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@tmp
>>> 4.0K box_maven-war-plugin_master-QSYAUU4F27NQH3VKQKYR34QIY6P36LH6ACXOADA2HDWCZVVU4TYA@tmp
>>> 155M    brooklyn-library-master-docker
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535791870056
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535929381454
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535991522759
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536015781206
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536050912838
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536188581290
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536350369118
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536361380993
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536447781251
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536534181007
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536620581304
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536706981637
>>> 451M    brooklyn-master-build-docker_ws-cleanup_1536793381496
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536966181681
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1536974386104
>>> 733M    brooklyn-master-build-docker_ws-cleanup_1537055864223
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537079895664
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537087960702
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537092848128
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537116168718
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537138981521
>>> 3.1G    brooklyn-master-build-docker_ws-cleanup_1537166290489
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537252717591
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537260673390
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537289677014
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537425657042
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537433463888
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537461822531
>>> 1.4G    Calcite-Master
>>> 170M _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q
>>> 8.0K _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q@tmp
>>> 2.8G Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA
>>> 4.0K Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA@tmp
>>> 171M Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ
>>> 4.0K Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ@tmp
>>> 60M Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A
>>> 4.0K Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A@tmp
>>> 1.9G Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ
>>> 4.0K Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ@tmp
>>> 2.5G Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ
>>> 4.0K Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ@tmp
>>> 3.3G    Camel.daily
>>> 4.0K    Camel.daily@tmp
>>> 2.4G Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q
>>> 4.0K Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q@tmp
>>> 2.3G Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ
>>> 4.0K Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ@tmp
>>> 810M    carbondata-master-spark-2.1
>>> 4.0K    carbondata-master-spark-2.1@tmp
>>> 584K    commons-geometry
>>> 4.0K CouchDB_master-FP4LRZTK2IWL4WCWS25JHGLJVPORIFX6ES6POVJARPVUTCZMU7SA@tmp
>>> 190M    CXF-Master-Analysis
>>> 188M    CXF-Master-JDK10
>>> 24M    CXF-Master-JDK9
>>> 1.7G    CXF-trunk-deploy
>>> 2.2G    CXF-Trunk-JDK18
>>> 74M    DeltaSpike Deploy
>>> 22M    DeltaSpike for Java EE
>>> 53M    DeltaSpike OWB 1.7.2
>>> 70M    DeltaSpike OWB 2.0.6
>>> 4.0K -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@2@tmp
>>> 4.0K -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@tmp
>>> 7.1M    Derby-JaCoCo
>>> 7.3M    dir-ldap-api-jdk8-linux SONAR
>>> 23M    dir-ldap-api-ubuntu-deploy
>>> 2.0G    dir-studio
>>> 2.7G    dir-studio-ui-tests-in-docker
>>> 1.3M    Empire-db reports
>>> 19M    Empire-db snapshot
>>> 4.0K en-artifact-resolver_master-TCRLMCKTTCPXV2IPJG6T7K54FQQY2O7AFHHEFILBSO6ZFSIUCGRQ@tmp
>>> 4.0K en-box_maven-mapping_master-VTBNHOZQB7UZ4ORPSWLQNN55JVWPXJBJNR54WLBERZKMDQGEPMXA@tmp
>>> 4.0K en-jxr_asfMavenTlpPlgnBuild-WCSOEZUCROETKKJCNPTYSFNDV52TA7MIFL63KMUYVDEPFRT5OK4Q@tmp
>>> 4.0K en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@2@tmp
>>> 4.0K en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@tmp
>>> 4.0K en-toolchains-plugin_master-JSJRH7N7WHA5OITGTMSQVCI6AVNUPHTOLBYXKJXWNGGNGCYHSF5A@tmp
>>> 287M g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA
>>> 4.0K g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA@tmp
>>> 419M    guacamole-rest-api-test
>>> 997M HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ
>>> 4.0K HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ@tmp
>>> 333M HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A
>>> 4.0K HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A@tmp
>>> 2.6G HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q
>>> 8.0K HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q@tmp
>>> 36M HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ
>>> 4.0K HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ@tmp
>>> 1.4G HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q
>>> 4.0K HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q@tmp
>>> 596K HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ
>>> 4.0K HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ@tmp
>>> 2.8G HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA
>>> 4.0K HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@tmp
>>> 538M    Hive-trunk
>>> 3.2G    incubator-netbeans-linux
>>> 744M    incubator-netbeans-linux-experiment
>>> 4.0K -info-reports-plugin_master-32Y3P743XLZN3V3HHV4NR6ZSZMFDM7Z5PXO3RSN6SA3U6NVU4ZTA@tmp
>>> 108K    infra-test-ant-ubuntu
>>> 12K    infra-test-docker-ubuntu
>>> 60K    infra-test-jdk-ubuntu
>>> 20K    infra-test-jdk-ubuntu-harmony
>>> 40K    infra-test-jdk-ubuntu-ibm
>>> 56K    infra-test-maven-ubuntu
>>> 12K    infra-test-snap-ubuntu
>>> 12K    infra-test-xvfb
>>> 1.4M    Ivy-Snapshot-Deploy
>>> 99M    Ivy-tests-ubuntu
>>> 16M    Ivy-tests-Windows
>>> 12M    Jackrabbit-trunk
>>> 4.0K javadoc-plugin_MJAVADOC-444-JPYTCQM73CF7UEOSYS6Z2MLTMHWVHYCB5QGVIGAH4MONP6SX7OHA@tmp
>>> 4.0K javadoc-plugin_MJAVADOC-516-Z27AS7O3YE3KP4HN2X3EMP56QPDDJR4P5EBKM2B5MKLXDNLXZ3DA@tmp
>>> 19M    Jena_Development_Test
>>> 731M    Jena_JDK11
>>> 264M    kafka-1.0-jdk7
>>> 290M    kafka-1.1-jdk7
>>> 501M    kafka-2.0-jdk8
>>> 628M    kafka-pr-jdk10-scala2.12
>>> 582M    kafka-pr-jdk8-scala2.11
>>> 523M    kafka-trunk-jdk8
>>> 60M    karaf-cave-master
>>> 4.8G    karaf-master
>>> 344M    Log4j 2 3.x
>>> 173M Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ
>>> 93M Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2
>>> 4.0K Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2@tmp
>>> 4.0K Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@tmp
>>> 4.0K ly-plugin_FIXING-JDK-ISSUES-IEX46HBDBJ6NRUCALBPK7SVRYXYL7WLKQAOOVWNB63FAXVHXQWMQ@tmp
>>> 11M maticModuleNameImprovements-5CTZJI7MCAG23DNAIMMOTYFEI4EKBOZ2AOP3EOG34Q46K3EBQVBA
>>> 4.0K _maven-archiver_MSHARED-721-QXH4Z3WHL2IJTFJJIFNELXYZM6YOEOOXEJCOQA7JK5JMIS42PNSA@tmp
>>> 4.0K maven-box_maven-jxr_JXR-110-R2OIXR7IG7PF3QUMPP2IV53ZAKEOGWXENXIP7G2LROL7YOEOZZTA@tmp
>>> 4.0K maven-box_maven-jxr_JXR-125-KFC5JZL777UAPG4UY6F434EV6BJ2ZRPG7HBO32YUS3UJ56NWROZQ@tmp
>>> 4.0K maven-box_maven-jxr_master-HV7BHOKN2YNGQ4SOGMW6QYUNHKCZDFS3I74XW3IMRQXBAUKY6MSQ@tmp
>>> 8.0K maven-box_maven_master-DVEN5KAF4FLE3WDK7SWFNYUY4YZ7X45JVWYY6FZT5ZEB3CLWABVQ
>>> 5.7M maven-box_maven_MNG-5868-TIXVUCFHOPHJ7LSBIS4NVVVTXHOVMMQO4W2BXVF455AQYGYZ26SA
>>> 8.0K maven-box_maven_MNG-6391-BFSVDMPDHLGZMTGMVMMLWPKG4K2P6CHHPIPVW27SUVBB6UQWV7YA
>>> 8.0K maven-box_maven_MNG-6475-LMWI4CBOH2JNDHX4PPHOWPHEASG5JKLTTZW3URJSHEBTPFA2447Q
>>> 4.0K maven-box_maven-osgi_master-537V4ALFOX2ZNQAIHHNQL5TC7LIKAHUUTBK4WE42PW2CUAN3GTBQ@tmp
>>> 4.0K maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@2@tmp
>>> 4.0K maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@tmp
>>> 4.0K maven-changes-plugin_master-O3PQ7WZZHAA2T5PBIGGQZFFXFD6PLMGLZHAF4U525J6NJFJA5IMQ@tmp
>>> 4.0K maven-install-plugin_master-3R4FSR6ITADJX4LXJM433BEETYXHTGVGRU6VEEEHH2PU3W5LNT6A@tmp
>>> 4.0K _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@2@tmp
>>> 4.0K _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@tmp
>>> 4.0K _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@2@tmp
>>> 4.0K _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@tmp
>>> 4.0K maven-reporting-exec_master-XY3GEBFQ42VNXZOS5MVD7PZKI6HQ7WOMNLQJ5IUZ7XSHVEI5U6RA@tmp
>>> 5.0G    Mesos-Buildbot
>>> 1.2M    Mesos-Reviewbot
>>> 4.0K mpiler-plugin_MCOMPILER-320-LDDLDH7HQQYXD6B2RHESP35HSEBJ26NA355LSJP2PHGG3DW43OPQ@tmp
>>> 4.0K m-publish-plugin_MSCMPUB-38-ULK6I7N2ZA3FU5JGEWCSL4ODBTZDTTC77NU7CUHQJKPNE7QVYS7A@tmp
>>> 4.0K n-archiver_MSHARED-668-test-WG4DQP6EJS2N73J3VGM2JH6SMRRHF477RDRLQRWF4WAHIMWMG5OA@tmp
>>> 4.0K n-box_maven-enforcer_master-KHI64AG3D4VZKUKLMNAYHKONI5AIQBLJ5ZKYGGSW7DAZVXCJBFHQ@tmp
>>> 4.0K n-box_maven-release_cleanup-YGG2KC5W5NBVLT77VULX2CHSYMVGRKDIZ7FJM54R7TMFDTDO6JYA@tmp
>>> 4.0K n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@2@tmp
>>> 4.0K n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@tmp
>>> 4.0K n-box_maven-wagon_WAGON-482-U5UQDLHLBEYZ6YKA2XFN2FQ74OK76EBRADZ4AK62MZWPJOQW2HDQ@tmp
>>> 4.0K n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@2@tmp
>>> 4.0K n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@tmp
>>> 4.0K n-box_maven-wagon_WAGON-527-KWYDRBGRPK7Q2YVCYHYJDNDOSEOHGQPJHV5AHJB4GNG66RKBHCBA@tmp
>>> 28K    Nemo
>>> 4.0K nfo-reports-plugin_MPIR-366-S464FI4ATSVOULE3N7MCKV5FYYG6GVHDD7OLISMYBGUEPLQM2UNA@tmp
>>> 5.7M _not-updated-MJAR-MCOMPILER-EYMH2UKFHCTFDWW4GZGG3AX4OJLGLPGZO6TAQGLBDBFX4K6CQ5GQ
>>> 4.0K n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@2@tmp
>>> 4.0K n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@tmp
>>> 4.0K n-scm-publish-plugin_master-2GNHI3CFRHSCHWWMDIRTSWMJIE3KK6DFWPSDNLSGKYY2ZVCACIZA@tmp
>>> 4.0K n-testing_MPLUGINTESTING-55-AWMCSLLBQGJRKNRZR27EZX2B5SVGADVMVK4FGA6YEL5S4OR3TOZQ@tmp
>>> 4.0K n-testing_MPLUGINTESTING-58-UONNAARVYXAHZ4RVLUBM5ODIM6A4TFB5CEC4ELRKMGU3KDYXAPIQ@tmp
>>> 800K nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA
>>> 4.0K nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA@tmp
>>> 343M    OpenWhisk-DockerHub
>>> 4.0K    OpenWhisk-DockerHub@tmp
>>> 4.0K ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@2@tmp
>>> 4.0K ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@tmp
>>> 4.0K ox_maven-fluido-skin_master-6RM5JATOOS6MTECVSVHV7YNCQIINRWOBDST2UCUIKNH64TYTME6Q@tmp
>>> 4.0K ox_maven-help-plugin_master-U66QJWH3TRYC4M2J55DEV6AO7VQ2JW77H5GHKBUOBKZTW32WDTTA@tmp
>>> 4.0K ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@2@tmp
>>> 4.0K ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@tmp
>>> 4.0K ox_maven-site-plugin_master-LZWAKOPRMO5JAZW5HRA2IQ56LLTFU3PLKDIE2JNFFXEGASBYKK3Q@tmp
>>> 667M    PDFBox-trunk
>>> 171M    PDFBox-Trunk-jdk10
>>> 52M    Phoenix-4.x-HBase-1.2
>>> 1.6G    Phoenix-4.x-HBase-1.3
>>> 6.7M    PreCommit-Admin
>>> 2.4G pulsar_precommit_integrationtests_ws-cleanup_1537313772888
>>> 4.4G pulsar_precommit_integrationtests_ws-cleanup_1537469839992
>>> 331M    Qpid-Broker-J-7.0.x-TestMatrix
>>> 331M    Qpid-Broker-J-Artefact-Release
>>> 339M    Qpid-Broker-J-JoramJMSTest
>>> 733M    Qpid-Broker-J-Python-Test
>>> 240M    Qpid-Broker-J-Test-IBMJDK1.8
>>> 12M    Qpid-Java-Java-6.1.x-TestMatrix
>>> 29M    Qpid-proton-c
>>> 1.3M    Qpid-proton-j
>>> 12M    ranger-0.7-snapshot-publish
>>> 256M    Ratis-Master
>>> 4.0K rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@2@tmp
>>> 4.0K rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@tmp
>>> 125M    samza-pr-checks
>>> 4.0K sembly-plugin_MASSEMBLY-892-AIWNXXLMVTBQ62GDGAZ5RR52AYYD2PS4MNOQBOW2K6A6AS4WHTGQ@tmp
>>> 50M    ServiceMix-Bundles
>>> 36M sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A
>>> 4.0K sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A@tmp
>>> 282M    Struts-2-3-JDK6
>>> 4.0K -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@2@tmp
>>> 4.0K -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@tmp
>>> 16M    Syncope-2_0_X-deploy
>>> 4.3G    Syncope-2_1_X-deploy
>>> 16M    Syncope-master-deploy
>>> 21M    Tamaya-Javadoc-Master
>>> 23M    Tamaya-Master-branch
>>> 74M    Tobago Trunk
>>> 81M    Tobago Trunk Sonar
>>> 4.0K    Tomcat-7.x-Maven
>>> 350M    trafficcontrol-master-build
>>> 4.0K    trafficcontrol-master-build@tmp
>>> 115M    trafficcontrol-PR-rat
>>> 41M    trafficcontrol-PR-trafficops-test
>>> 162M    trafficcontrol-traffic_ops-test
>>> 4.0K    trafficcontrol-traffic_ops-test@tmp
>>> 590M    unomi-master
>>> 4.0K ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@2@tmp
>>> 4.0K ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@tmp
>>> 4.0K ven-box_maven-surefire_1532-O6EC5ODCIKDKZL25QDLH6QFMFZD5LDHASAYEMDZSNREZV4N37X7A@tmp
>>> 4.0K ven-scripting-plugin_master-F7DFWWXZD7KAQGDPAQML3SYNOE2CUDOJDQGTJN7NKKPXY3SWUHCQ@tmp
>>> 4.0K ven-site-plugin_MNGSITE-332-4LYPVCGK77QO5G6UKI6IR7JSEN5EMK7EVKG6TK2D3VYBAHH6QPBA@tmp
>>> 4.0K x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@2@tmp
>>> 4.0K x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@tmp
>>> 4.0K x_maven-invoker_MSHARED-752-O2LI4W4PL6ARSMCGVOOVX6GZJCOYVGXMTNU5KTGD65M6CNMTDSRQ@tmp
>>> 4.0K x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@2@tmp
>>> 4.0K x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@tmp
>>> 4.0K x_maven-jlink-plugin_master-TWLBBVLGC7P5GUJSJEP6SD46MW2WP64SO5MED7OJBUOM66RIHIJQ@tmp
>>> 4.0K x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@2@tmp
>>> 4.0K x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@tmp
>>> 4.0K x_maven-release_IMPROVE_POM-KZP6VA3ZSGGYNTVVOCGLUBHYECJZQ34VIS6H2U5HNYVKLJFCFQKQ@tmp
>>> 4.0K x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@2@tmp
>>> 4.0K x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@tmp
>>> 4.0K x_maven-stage-plugin_master-DUIB3VWMFIP4K7X4DPTLYSUTRVJKIKKQPIMXJ3UK6V5VUGFJHXHQ@tmp
>>> 4.0K x_maven-wagon_wagon-scm-git-JFLMMYBLUJZHZUM4KST663ZOB5W6QQB2OF5JLK4YVV3OMUVRSJDA@tmp
>>> 340M    Yetus - OWASP dependency-check cache
>>> 119M    zeppelin-pull-request
>>> 93M    ZooKeeper-Find-Flaky-Tests
>>> 465M    ZooKeeper-trunk-openjdk10
>>> 157M    ZooKeeper-trunk-owasp
>>> 119M    ZooKeeper_UT_Stress
>> 
>

Re: Empire-db build problems <-- brooklyn-master-build-docker_ws-cleanup workspace left behind

Posted by Aled Sage <al...@cloudsoftcorp.com>.

// removing builds@apache.org to save them this chatter

Hi all,

I've reconfigured the builds for brooklyn-library so that it does not 
bind-mount [1] the jenkins ${HOME}/.m2 directory from the jenkins host, 
when running the build/tests in the container.

I've also switched it to use Jenkins pipeline, so it actually uses the 
Jenkinsfile [2] that Thomas added to the repo back in November! There 
are two new builds configured:

https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker-pipeline/
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-pull-requests-pipeline

Once we're happy with these, we should disable (or perhaps delete) the 
old build configurations:

https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-pull-requests/

My next steps are:

 1. Disable the old build configurations.
 2. Reconfigure the other brooklyn repos (and build configurations) to
    follow the pattern used in brooklyn-library.
 3. Monitor the builds, to see if this is enough to fix the problems we
    were seeing.
    (however, we still bind-mount the writeable workspace directory in
    the container, so there is still room for improvement I think).

Please shout if you have any comments, or think we should take this in a 
different direction instead.

Aled

[1] https://docs.docker.com/storage/bind-mounts/#use-a-read-only-bind-mount
[2] https://github.com/apache/brooklyn-library/blob/master/Jenkinsfile


On 24/09/2018 15:20, Aled Sage wrote:
> Hi Chris, all,
>
> Thomas has just pointed me at 
> https://issues.apache.org/jira/browse/INFRA-16417, in which you recommend:
>
>     "The problem is the bind mount for .m2 and permissions conflicts
>     between the docker
>     image and jenkins. Please don't use a bind mount for filesystems
>     you intend to write to.
>     It will make the build run a bit longer since it has to download
>     the artifacts within the
>     container, but it will prevent you from having problems and
>     conflicts with the
>     jenkins .m2 directory."
>
> Apologies for us not running with this advice previously - I'm looking 
> into how we could do this.
>
> Are there any good example configurations of other Apache projects 
> building and running tests (ideally using maven) in docker containers, 
> on apache jenkins slaves?
>
> Aled
>
>
> On 21/09/2018 10:03, Aled Sage wrote:
>> Hi all,
>>
>> Do any brooklyn folk know what the 
>> `brooklyn-master-build-docker_ws-cleanup_.*` workspaces are for, and 
>> why they are not being cleaned up?
>>
>> It sounds like this bug in ws-cleanup-plugin [1]. It was fixed, but 
>> then folk reported that they were still experiencing the same problem 
>> [2].
>>
>> A workaround described in [3] is:
>>
>>   ws-cleanup is leaking workspaces because the dynamic nodes are 
>> unavailable at the time
>>   the dispose is ran. I've installed a cron which is run every 1/2/h 
>> which is removing
>>   ws-cleanup folders that aren't modified the last hour on ovh docker 
>> hosts.
>>
>> I'm not sure what we could do on the Brooklyn side.
>>
>> ---
>> Looking in the console output of our brooklyn-master-build-docker 
>> jobs, I see #700 [4] says at the very start:
>>
>>   Building remotely on H24 (ubuntu xenial) in workspace 
>> /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker
>>   [WS-CLEANUP] Deleting project workspace...
>>   [WS-CLEANUP] Done
>>
>> In contrast, builds #702 [5] doesn't have the line `[WS-CLEANUP] 
>> Done`. Does that mean it will have left behind one of these 
>> directories perhaps?
>>
>> Looking at [6], it looks like we only keep a history of 5 builds in 
>> jenkins. We therefore should really not have 28 workspace directories 
>> on the server H24!
>>
>> If it might be a problem with ws-cleanup having permissions to delete 
>> the workspace, could it be related to the way we mount the workspace 
>> as a volume in the docker container?
>>
>>   docker run -i --rm -v /home/jenkins:/usr/home
>>       -v 
>> /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker/.m2:/root/.m2 
>> \
>>       -v 
>> /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker:/usr/build 
>> \
>> -w /usr/build brooklyn:latest mvn clean install -s 
>> /usr/home/.m2/settings.xml
>>
>> Aled
>>
>> [1] https://issues.jenkins-ci.org/browse/JENKINS-24824
>> [2] https://issues.jenkins-ci.org/browse/JENKINS-53579
>> [3] https://jira.nuxeo.com/browse/NXBT-2205
>> [4] 
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
>> [5] 
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
>> [6] https://builds.apache.org/job/brooklyn-master-build-docker/
>>
>>
>> On 20/09/2018 20:20, Chris Lambertus wrote:
>>>
>>>
>>>
>>>
>>>> On Sep 20, 2018, at 11:51 AM, Dave Brondsema <da...@brondsema.net> 
>>>> wrote:
>>>>
>>>> Moving to builds@
>>>>
>>>> I'm seeing this type of disk error on Allura builds that have used 
>>>> H27 and H24
>>>> today.
>>> Here’s the raw usage output from H24. Not sure what’s going on with 
>>> those Brooklyn jobs. CC to dev@brooklyn so they can have a look.
>>>
>>>
>>>
>>> root@asf924:/home/jenkins/jenkins-slave/workspace# du -sh -- *
>>> 4.3G    ActiveMQ-Java8
>>> 2.5G    ActiveMQ-Java8-All-UnitTests
>>> 1.2G    ActiveMQ-PR-Build
>>> 4.5G    Ambari-branch-2.7
>>> 8.0G    Ambari-trunk-Commit
>>> 127M    Ant-Build-Matrix-1.9.x-Linux
>>> 129M    Ant-Build-Matrix-master-Linux
>>> 88M    Ant-Master SonarQube
>>> 4.7M    Apache%20Jackrabbit%20Oak%20matrix
>>> 15G    Apache Jackrabbit Oak matrix
>>> 12M    Aries jax-rs whiteboard
>>> 14M    Aries-rsa master deploy
>>> 816M    aurora-packaging-nightly
>>> 4.0K 
>>> aven-assembly-plugin_master-SP4DXUL4GVN3Q5KJHI5ED6WPWUEC2L2PUW2RCT2SFVGTCXHUIFDQ@tmp
>>> 4.0K 
>>> aven-box_maven-doxia_master-4SHBBM33BQC2SGN65AA3AW6GPXIK6RA462YQND5AN3YPZ5U2VGSQ@tmp
>>> 4.0K 
>>> aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@2@tmp
>>> 4.0K 
>>> aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@tmp
>>> 4.0K 
>>> aven-enforcer_MENFORCER-312-H6BWWMKU3O4RIVLVOSDDUVD7CT73H5PSEGRJICFTH5FH5BR7L3XQ@tmp
>>> 122M    Bigtop-trunk-testartifacts
>>> 924M    bookkeeper_postcommit_master_java8
>>> 261M    bookkeeper_precommit_bookie_tests
>>> 302M    bookkeeper_precommit_client_tests
>>> 754M    bookkeeper_precommit_pullrequest_java8
>>> 758M    bookkeeper_precommit_pullrequest_java9
>>> 272M    bookkeeper_precommit_pullrequest_validation
>>> 400M    bookkeeper_precommit_pullrequest_validation@2
>>> 334M    bookkeeper_precommit_remaining_tests
>>> 257M    bookkeeper_precommit_replication_tests
>>> 259M    bookkeeper_precommit_tls_tests
>>> 907M    bookkeeper_release_branch_48_integrationtests
>>> 886M    bookkeeper_release_branch_48_java8
>>> 4.0K 
>>> box_maven-ant-plugin_master-N743QKOWXTZ36LZCK2YK3A6N4ZY3UG7X5RV3MH7PEH646A2IMDHA@tmp
>>> 4.0K 
>>> -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@2@tmp
>>> 4.0K 
>>> -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@tmp
>>> 4.0K 
>>> box_maven-ejb-plugin_master-D5MGOBWWYZCBR23TA7FG7MUJQK233O27B5TSPA7YUOAEWLP5Z72Q@tmp
>>> 4.0K 
>>> -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@2@tmp
>>> 4.0K 
>>> -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@tmp
>>> 5.7M 
>>> -box_maven_pre-reset-master-AF72GWR2EBH4F7AYOLQPRDHKOEZISK7V6BEF33K5DPJKW2VE4ATA
>>> 4.0K 
>>> box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@2@tmp
>>> 4.0K 
>>> box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@tmp
>>> 4.0K 
>>> -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@2@tmp
>>> 4.0K 
>>> -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@tmp
>>> 4.0K 
>>> box_maven-war-plugin_master-QSYAUU4F27NQH3VKQKYR34QIY6P36LH6ACXOADA2HDWCZVVU4TYA@tmp
>>> 155M    brooklyn-library-master-docker
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535791870056
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535929381454
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535991522759
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536015781206
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536050912838
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536188581290
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536350369118
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536361380993
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536447781251
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536534181007
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536620581304
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536706981637
>>> 451M    brooklyn-master-build-docker_ws-cleanup_1536793381496
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536966181681
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1536974386104
>>> 733M    brooklyn-master-build-docker_ws-cleanup_1537055864223
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537079895664
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537087960702
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537092848128
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537116168718
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537138981521
>>> 3.1G    brooklyn-master-build-docker_ws-cleanup_1537166290489
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537252717591
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537260673390
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537289677014
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537425657042
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537433463888
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537461822531
>>> 1.4G    Calcite-Master
>>> 170M 
>>> _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q
>>> 8.0K 
>>> _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q@tmp
>>> 2.8G Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA
>>> 4.0K 
>>> Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA@tmp
>>> 171M 
>>> Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ
>>> 4.0K 
>>> Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ@tmp
>>> 60M 
>>> Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A
>>> 4.0K 
>>> Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A@tmp
>>> 1.9G 
>>> Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ
>>> 4.0K 
>>> Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ@tmp
>>> 2.5G 
>>> Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ
>>> 4.0K 
>>> Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ@tmp
>>> 3.3G    Camel.daily
>>> 4.0K    Camel.daily@tmp
>>> 2.4G Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q
>>> 4.0K 
>>> Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q@tmp
>>> 2.3G Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ
>>> 4.0K 
>>> Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ@tmp
>>> 810M    carbondata-master-spark-2.1
>>> 4.0K    carbondata-master-spark-2.1@tmp
>>> 584K    commons-geometry
>>> 4.0K 
>>> CouchDB_master-FP4LRZTK2IWL4WCWS25JHGLJVPORIFX6ES6POVJARPVUTCZMU7SA@tmp
>>> 190M    CXF-Master-Analysis
>>> 188M    CXF-Master-JDK10
>>> 24M    CXF-Master-JDK9
>>> 1.7G    CXF-trunk-deploy
>>> 2.2G    CXF-Trunk-JDK18
>>> 74M    DeltaSpike Deploy
>>> 22M    DeltaSpike for Java EE
>>> 53M    DeltaSpike OWB 1.7.2
>>> 70M    DeltaSpike OWB 2.0.6
>>> 4.0K 
>>> -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@2@tmp
>>> 4.0K 
>>> -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@tmp
>>> 7.1M    Derby-JaCoCo
>>> 7.3M    dir-ldap-api-jdk8-linux SONAR
>>> 23M    dir-ldap-api-ubuntu-deploy
>>> 2.0G    dir-studio
>>> 2.7G    dir-studio-ui-tests-in-docker
>>> 1.3M    Empire-db reports
>>> 19M    Empire-db snapshot
>>> 4.0K 
>>> en-artifact-resolver_master-TCRLMCKTTCPXV2IPJG6T7K54FQQY2O7AFHHEFILBSO6ZFSIUCGRQ@tmp
>>> 4.0K 
>>> en-box_maven-mapping_master-VTBNHOZQB7UZ4ORPSWLQNN55JVWPXJBJNR54WLBERZKMDQGEPMXA@tmp
>>> 4.0K 
>>> en-jxr_asfMavenTlpPlgnBuild-WCSOEZUCROETKKJCNPTYSFNDV52TA7MIFL63KMUYVDEPFRT5OK4Q@tmp
>>> 4.0K 
>>> en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@2@tmp
>>> 4.0K 
>>> en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@tmp
>>> 4.0K 
>>> en-toolchains-plugin_master-JSJRH7N7WHA5OITGTMSQVCI6AVNUPHTOLBYXKJXWNGGNGCYHSF5A@tmp
>>> 287M 
>>> g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA
>>> 4.0K 
>>> g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA@tmp
>>> 419M    guacamole-rest-api-test
>>> 997M 
>>> HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ
>>> 4.0K 
>>> HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ@tmp
>>> 333M 
>>> HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A
>>> 4.0K 
>>> HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A@tmp
>>> 2.6G 
>>> HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q
>>> 8.0K 
>>> HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q@tmp
>>> 36M 
>>> HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ
>>> 4.0K 
>>> HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ@tmp
>>> 1.4G 
>>> HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q
>>> 4.0K 
>>> HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q@tmp
>>> 596K 
>>> HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ
>>> 4.0K 
>>> HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ@tmp
>>> 2.8G 
>>> HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA
>>> 4.0K 
>>> HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@tmp
>>> 538M    Hive-trunk
>>> 3.2G    incubator-netbeans-linux
>>> 744M    incubator-netbeans-linux-experiment
>>> 4.0K 
>>> -info-reports-plugin_master-32Y3P743XLZN3V3HHV4NR6ZSZMFDM7Z5PXO3RSN6SA3U6NVU4ZTA@tmp
>>> 108K    infra-test-ant-ubuntu
>>> 12K    infra-test-docker-ubuntu
>>> 60K    infra-test-jdk-ubuntu
>>> 20K    infra-test-jdk-ubuntu-harmony
>>> 40K    infra-test-jdk-ubuntu-ibm
>>> 56K    infra-test-maven-ubuntu
>>> 12K    infra-test-snap-ubuntu
>>> 12K    infra-test-xvfb
>>> 1.4M    Ivy-Snapshot-Deploy
>>> 99M    Ivy-tests-ubuntu
>>> 16M    Ivy-tests-Windows
>>> 12M    Jackrabbit-trunk
>>> 4.0K 
>>> javadoc-plugin_MJAVADOC-444-JPYTCQM73CF7UEOSYS6Z2MLTMHWVHYCB5QGVIGAH4MONP6SX7OHA@tmp
>>> 4.0K 
>>> javadoc-plugin_MJAVADOC-516-Z27AS7O3YE3KP4HN2X3EMP56QPDDJR4P5EBKM2B5MKLXDNLXZ3DA@tmp
>>> 19M    Jena_Development_Test
>>> 731M    Jena_JDK11
>>> 264M    kafka-1.0-jdk7
>>> 290M    kafka-1.1-jdk7
>>> 501M    kafka-2.0-jdk8
>>> 628M    kafka-pr-jdk10-scala2.12
>>> 582M    kafka-pr-jdk8-scala2.11
>>> 523M    kafka-trunk-jdk8
>>> 60M    karaf-cave-master
>>> 4.8G    karaf-master
>>> 344M    Log4j 2 3.x
>>> 173M 
>>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ
>>> 93M 
>>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2
>>> 4.0K 
>>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2@tmp
>>> 4.0K 
>>> Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@tmp
>>> 4.0K 
>>> ly-plugin_FIXING-JDK-ISSUES-IEX46HBDBJ6NRUCALBPK7SVRYXYL7WLKQAOOVWNB63FAXVHXQWMQ@tmp
>>> 11M 
>>> maticModuleNameImprovements-5CTZJI7MCAG23DNAIMMOTYFEI4EKBOZ2AOP3EOG34Q46K3EBQVBA
>>> 4.0K 
>>> _maven-archiver_MSHARED-721-QXH4Z3WHL2IJTFJJIFNELXYZM6YOEOOXEJCOQA7JK5JMIS42PNSA@tmp
>>> 4.0K 
>>> maven-box_maven-jxr_JXR-110-R2OIXR7IG7PF3QUMPP2IV53ZAKEOGWXENXIP7G2LROL7YOEOZZTA@tmp
>>> 4.0K 
>>> maven-box_maven-jxr_JXR-125-KFC5JZL777UAPG4UY6F434EV6BJ2ZRPG7HBO32YUS3UJ56NWROZQ@tmp
>>> 4.0K 
>>> maven-box_maven-jxr_master-HV7BHOKN2YNGQ4SOGMW6QYUNHKCZDFS3I74XW3IMRQXBAUKY6MSQ@tmp
>>> 8.0K 
>>> maven-box_maven_master-DVEN5KAF4FLE3WDK7SWFNYUY4YZ7X45JVWYY6FZT5ZEB3CLWABVQ
>>> 5.7M 
>>> maven-box_maven_MNG-5868-TIXVUCFHOPHJ7LSBIS4NVVVTXHOVMMQO4W2BXVF455AQYGYZ26SA
>>> 8.0K 
>>> maven-box_maven_MNG-6391-BFSVDMPDHLGZMTGMVMMLWPKG4K2P6CHHPIPVW27SUVBB6UQWV7YA
>>> 8.0K 
>>> maven-box_maven_MNG-6475-LMWI4CBOH2JNDHX4PPHOWPHEASG5JKLTTZW3URJSHEBTPFA2447Q
>>> 4.0K 
>>> maven-box_maven-osgi_master-537V4ALFOX2ZNQAIHHNQL5TC7LIKAHUUTBK4WE42PW2CUAN3GTBQ@tmp
>>> 4.0K 
>>> maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@2@tmp
>>> 4.0K 
>>> maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@tmp
>>> 4.0K 
>>> maven-changes-plugin_master-O3PQ7WZZHAA2T5PBIGGQZFFXFD6PLMGLZHAF4U525J6NJFJA5IMQ@tmp
>>> 4.0K 
>>> maven-install-plugin_master-3R4FSR6ITADJX4LXJM433BEETYXHTGVGRU6VEEEHH2PU3W5LNT6A@tmp
>>> 4.0K 
>>> _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@2@tmp
>>> 4.0K 
>>> _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@tmp
>>> 4.0K 
>>> _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@2@tmp
>>> 4.0K 
>>> _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@tmp
>>> 4.0K 
>>> maven-reporting-exec_master-XY3GEBFQ42VNXZOS5MVD7PZKI6HQ7WOMNLQJ5IUZ7XSHVEI5U6RA@tmp
>>> 5.0G    Mesos-Buildbot
>>> 1.2M    Mesos-Reviewbot
>>> 4.0K 
>>> mpiler-plugin_MCOMPILER-320-LDDLDH7HQQYXD6B2RHESP35HSEBJ26NA355LSJP2PHGG3DW43OPQ@tmp
>>> 4.0K 
>>> m-publish-plugin_MSCMPUB-38-ULK6I7N2ZA3FU5JGEWCSL4ODBTZDTTC77NU7CUHQJKPNE7QVYS7A@tmp
>>> 4.0K 
>>> n-archiver_MSHARED-668-test-WG4DQP6EJS2N73J3VGM2JH6SMRRHF477RDRLQRWF4WAHIMWMG5OA@tmp
>>> 4.0K 
>>> n-box_maven-enforcer_master-KHI64AG3D4VZKUKLMNAYHKONI5AIQBLJ5ZKYGGSW7DAZVXCJBFHQ@tmp
>>> 4.0K 
>>> n-box_maven-release_cleanup-YGG2KC5W5NBVLT77VULX2CHSYMVGRKDIZ7FJM54R7TMFDTDO6JYA@tmp
>>> 4.0K 
>>> n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@2@tmp
>>> 4.0K 
>>> n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@tmp
>>> 4.0K 
>>> n-box_maven-wagon_WAGON-482-U5UQDLHLBEYZ6YKA2XFN2FQ74OK76EBRADZ4AK62MZWPJOQW2HDQ@tmp
>>> 4.0K 
>>> n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@2@tmp
>>> 4.0K 
>>> n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@tmp
>>> 4.0K 
>>> n-box_maven-wagon_WAGON-527-KWYDRBGRPK7Q2YVCYHYJDNDOSEOHGQPJHV5AHJB4GNG66RKBHCBA@tmp
>>> 28K    Nemo
>>> 4.0K 
>>> nfo-reports-plugin_MPIR-366-S464FI4ATSVOULE3N7MCKV5FYYG6GVHDD7OLISMYBGUEPLQM2UNA@tmp
>>> 5.7M 
>>> _not-updated-MJAR-MCOMPILER-EYMH2UKFHCTFDWW4GZGG3AX4OJLGLPGZO6TAQGLBDBFX4K6CQ5GQ
>>> 4.0K 
>>> n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@2@tmp
>>> 4.0K 
>>> n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@tmp
>>> 4.0K 
>>> n-scm-publish-plugin_master-2GNHI3CFRHSCHWWMDIRTSWMJIE3KK6DFWPSDNLSGKYY2ZVCACIZA@tmp
>>> 4.0K 
>>> n-testing_MPLUGINTESTING-55-AWMCSLLBQGJRKNRZR27EZX2B5SVGADVMVK4FGA6YEL5S4OR3TOZQ@tmp
>>> 4.0K 
>>> n-testing_MPLUGINTESTING-58-UONNAARVYXAHZ4RVLUBM5ODIM6A4TFB5CEC4ELRKMGU3KDYXAPIQ@tmp
>>> 800K 
>>> nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA
>>> 4.0K 
>>> nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA@tmp
>>> 343M    OpenWhisk-DockerHub
>>> 4.0K    OpenWhisk-DockerHub@tmp
>>> 4.0K 
>>> ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@2@tmp
>>> 4.0K 
>>> ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@tmp
>>> 4.0K 
>>> ox_maven-fluido-skin_master-6RM5JATOOS6MTECVSVHV7YNCQIINRWOBDST2UCUIKNH64TYTME6Q@tmp
>>> 4.0K 
>>> ox_maven-help-plugin_master-U66QJWH3TRYC4M2J55DEV6AO7VQ2JW77H5GHKBUOBKZTW32WDTTA@tmp
>>> 4.0K 
>>> ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@2@tmp
>>> 4.0K 
>>> ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@tmp
>>> 4.0K 
>>> ox_maven-site-plugin_master-LZWAKOPRMO5JAZW5HRA2IQ56LLTFU3PLKDIE2JNFFXEGASBYKK3Q@tmp
>>> 667M    PDFBox-trunk
>>> 171M    PDFBox-Trunk-jdk10
>>> 52M    Phoenix-4.x-HBase-1.2
>>> 1.6G    Phoenix-4.x-HBase-1.3
>>> 6.7M    PreCommit-Admin
>>> 2.4G pulsar_precommit_integrationtests_ws-cleanup_1537313772888
>>> 4.4G pulsar_precommit_integrationtests_ws-cleanup_1537469839992
>>> 331M    Qpid-Broker-J-7.0.x-TestMatrix
>>> 331M    Qpid-Broker-J-Artefact-Release
>>> 339M    Qpid-Broker-J-JoramJMSTest
>>> 733M    Qpid-Broker-J-Python-Test
>>> 240M    Qpid-Broker-J-Test-IBMJDK1.8
>>> 12M    Qpid-Java-Java-6.1.x-TestMatrix
>>> 29M    Qpid-proton-c
>>> 1.3M    Qpid-proton-j
>>> 12M    ranger-0.7-snapshot-publish
>>> 256M    Ratis-Master
>>> 4.0K 
>>> rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@2@tmp
>>> 4.0K 
>>> rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@tmp
>>> 125M    samza-pr-checks
>>> 4.0K 
>>> sembly-plugin_MASSEMBLY-892-AIWNXXLMVTBQ62GDGAZ5RR52AYYD2PS4MNOQBOW2K6A6AS4WHTGQ@tmp
>>> 50M    ServiceMix-Bundles
>>> 36M 
>>> sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A
>>> 4.0K 
>>> sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A@tmp
>>> 282M    Struts-2-3-JDK6
>>> 4.0K 
>>> -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@2@tmp
>>> 4.0K 
>>> -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@tmp
>>> 16M    Syncope-2_0_X-deploy
>>> 4.3G    Syncope-2_1_X-deploy
>>> 16M    Syncope-master-deploy
>>> 21M    Tamaya-Javadoc-Master
>>> 23M    Tamaya-Master-branch
>>> 74M    Tobago Trunk
>>> 81M    Tobago Trunk Sonar
>>> 4.0K    Tomcat-7.x-Maven
>>> 350M    trafficcontrol-master-build
>>> 4.0K    trafficcontrol-master-build@tmp
>>> 115M    trafficcontrol-PR-rat
>>> 41M    trafficcontrol-PR-trafficops-test
>>> 162M    trafficcontrol-traffic_ops-test
>>> 4.0K    trafficcontrol-traffic_ops-test@tmp
>>> 590M    unomi-master
>>> 4.0K 
>>> ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@2@tmp
>>> 4.0K 
>>> ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@tmp
>>> 4.0K 
>>> ven-box_maven-surefire_1532-O6EC5ODCIKDKZL25QDLH6QFMFZD5LDHASAYEMDZSNREZV4N37X7A@tmp
>>> 4.0K 
>>> ven-scripting-plugin_master-F7DFWWXZD7KAQGDPAQML3SYNOE2CUDOJDQGTJN7NKKPXY3SWUHCQ@tmp
>>> 4.0K 
>>> ven-site-plugin_MNGSITE-332-4LYPVCGK77QO5G6UKI6IR7JSEN5EMK7EVKG6TK2D3VYBAHH6QPBA@tmp
>>> 4.0K 
>>> x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@2@tmp
>>> 4.0K 
>>> x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@tmp
>>> 4.0K 
>>> x_maven-invoker_MSHARED-752-O2LI4W4PL6ARSMCGVOOVX6GZJCOYVGXMTNU5KTGD65M6CNMTDSRQ@tmp
>>> 4.0K 
>>> x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@2@tmp
>>> 4.0K 
>>> x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@tmp
>>> 4.0K 
>>> x_maven-jlink-plugin_master-TWLBBVLGC7P5GUJSJEP6SD46MW2WP64SO5MED7OJBUOM66RIHIJQ@tmp
>>> 4.0K 
>>> x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@2@tmp
>>> 4.0K 
>>> x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@tmp
>>> 4.0K 
>>> x_maven-release_IMPROVE_POM-KZP6VA3ZSGGYNTVVOCGLUBHYECJZQ34VIS6H2U5HNYVKLJFCFQKQ@tmp
>>> 4.0K 
>>> x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@2@tmp
>>> 4.0K 
>>> x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@tmp
>>> 4.0K 
>>> x_maven-stage-plugin_master-DUIB3VWMFIP4K7X4DPTLYSUTRVJKIKKQPIMXJ3UK6V5VUGFJHXHQ@tmp
>>> 4.0K 
>>> x_maven-wagon_wagon-scm-git-JFLMMYBLUJZHZUM4KST663ZOB5W6QQB2OF5JLK4YVV3OMUVRSJDA@tmp
>>> 340M    Yetus - OWASP dependency-check cache
>>> 119M    zeppelin-pull-request
>>> 93M    ZooKeeper-Find-Flaky-Tests
>>> 465M    ZooKeeper-trunk-openjdk10
>>> 157M    ZooKeeper-trunk-owasp
>>> 119M    ZooKeeper_UT_Stress
>>
>

Re: Empire-db build problems <-- brooklyn-master-build-docker_ws-cleanup workspace left behind

Posted by Allen Wittenauer <aw...@effectivemachines.com.INVALID>.

This is pretty much what Apache Yetus was built to do. (I’d go into more depth but about to board a plane. ) :D

Sent from my ☎️

> On Sep 24, 2018, at 3:20 PM, Aled Sage <al...@cloudsoftcorp.com> wrote:
> 
> Hi Chris, all,
> 
> Thomas has just pointed me at https://issues.apache.org/jira/browse/INFRA-16417, in which you recommend:
> 
>   "The problem is the bind mount for .m2 and permissions conflicts
>   between the docker
>   image and jenkins. Please don't use a bind mount for filesystems you
>   intend to write to.
>   It will make the build run a bit longer since it has to download the
>   artifacts within the
>   container, but it will prevent you from having problems and
>   conflicts with the
>   jenkins .m2 directory."
> 
> Apologies for us not running with this advice previously - I'm looking into how we could do this.
> 
> Are there any good example configurations of other Apache projects building and running tests (ideally using maven) in docker containers, on apache jenkins slaves?
> 
> Aled
> 
> 
>> On 21/09/2018 10:03, Aled Sage wrote:
>> Hi all,
>> 
>> Do any brooklyn folk know what the `brooklyn-master-build-docker_ws-cleanup_.*` workspaces are for, and why they are not being cleaned up?
>> 
>> It sounds like this bug in ws-cleanup-plugin [1]. It was fixed, but then folk reported that they were still experiencing the same problem [2].
>> 
>> A workaround described in [3] is:
>> 
>>   ws-cleanup is leaking workspaces because the dynamic nodes are unavailable at the time
>>   the dispose is ran. I've installed a cron which is run every 1/2/h which is removing
>>   ws-cleanup folders that aren't modified the last hour on ovh docker hosts.
>> 
>> I'm not sure what we could do on the Brooklyn side.
>> 
>> ---
>> Looking in the console output of our brooklyn-master-build-docker jobs, I see #700 [4] says at the very start:
>> 
>>   Building remotely on H24 (ubuntu xenial) in workspace /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker
>>   [WS-CLEANUP] Deleting project workspace...
>>   [WS-CLEANUP] Done
>> 
>> In contrast, builds #702 [5] doesn't have the line `[WS-CLEANUP] Done`. Does that mean it will have left behind one of these directories perhaps?
>> 
>> Looking at [6], it looks like we only keep a history of 5 builds in jenkins. We therefore should really not have 28 workspace directories on the server H24!
>> 
>> If it might be a problem with ws-cleanup having permissions to delete the workspace, could it be related to the way we mount the workspace as a volume in the docker container?
>> 
>>   docker run -i --rm -v /home/jenkins:/usr/home
>>       -v /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker/.m2:/root/.m2 \
>>       -v /home/jenkins/jenkins-slave/workspace/brooklyn-master-build-docker:/usr/build \
>> -w /usr/build brooklyn:latest mvn clean install -s /usr/home/.m2/settings.xml
>> 
>> Aled
>> 
>> [1] https://issues.jenkins-ci.org/browse/JENKINS-24824
>> [2] https://issues.jenkins-ci.org/browse/JENKINS-53579
>> [3] https://jira.nuxeo.com/browse/NXBT-2205
>> [4] https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
>> [5] https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/700/consoleFull
>> [6] https://builds.apache.org/job/brooklyn-master-build-docker/
>> 
>> 
>>> On 20/09/2018 20:20, Chris Lambertus wrote:
>>> 
>>> 
>>> 
>>> 
>>>> On Sep 20, 2018, at 11:51 AM, Dave Brondsema <da...@brondsema.net> wrote:
>>>> 
>>>> Moving to builds@
>>>> 
>>>> I'm seeing this type of disk error on Allura builds that have used H27 and H24
>>>> today.
>>> Here’s the raw usage output from H24. Not sure what’s going on with those Brooklyn jobs. CC to dev@brooklyn so they can have a look.
>>> 
>>> 
>>> 
>>> root@asf924:/home/jenkins/jenkins-slave/workspace# du -sh -- *
>>> 4.3G    ActiveMQ-Java8
>>> 2.5G    ActiveMQ-Java8-All-UnitTests
>>> 1.2G    ActiveMQ-PR-Build
>>> 4.5G    Ambari-branch-2.7
>>> 8.0G    Ambari-trunk-Commit
>>> 127M    Ant-Build-Matrix-1.9.x-Linux
>>> 129M    Ant-Build-Matrix-master-Linux
>>> 88M    Ant-Master SonarQube
>>> 4.7M    Apache%20Jackrabbit%20Oak%20matrix
>>> 15G    Apache Jackrabbit Oak matrix
>>> 12M    Aries jax-rs whiteboard
>>> 14M    Aries-rsa master deploy
>>> 816M    aurora-packaging-nightly
>>> 4.0K aven-assembly-plugin_master-SP4DXUL4GVN3Q5KJHI5ED6WPWUEC2L2PUW2RCT2SFVGTCXHUIFDQ@tmp
>>> 4.0K aven-box_maven-doxia_master-4SHBBM33BQC2SGN65AA3AW6GPXIK6RA462YQND5AN3YPZ5U2VGSQ@tmp
>>> 4.0K aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@2@tmp
>>> 4.0K aven-enforcer_MENFORCER-288-GL5B4ERVFARMDDY5GUJ32NFJSOP7ZR6QLATDJG7AYHOHJMZGS6NA@tmp
>>> 4.0K aven-enforcer_MENFORCER-312-H6BWWMKU3O4RIVLVOSDDUVD7CT73H5PSEGRJICFTH5FH5BR7L3XQ@tmp
>>> 122M    Bigtop-trunk-testartifacts
>>> 924M    bookkeeper_postcommit_master_java8
>>> 261M    bookkeeper_precommit_bookie_tests
>>> 302M    bookkeeper_precommit_client_tests
>>> 754M    bookkeeper_precommit_pullrequest_java8
>>> 758M    bookkeeper_precommit_pullrequest_java9
>>> 272M    bookkeeper_precommit_pullrequest_validation
>>> 400M    bookkeeper_precommit_pullrequest_validation@2
>>> 334M    bookkeeper_precommit_remaining_tests
>>> 257M    bookkeeper_precommit_replication_tests
>>> 259M    bookkeeper_precommit_tls_tests
>>> 907M    bookkeeper_release_branch_48_integrationtests
>>> 886M    bookkeeper_release_branch_48_java8
>>> 4.0K box_maven-ant-plugin_master-N743QKOWXTZ36LZCK2YK3A6N4ZY3UG7X5RV3MH7PEH646A2IMDHA@tmp
>>> 4.0K -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@2@tmp
>>> 4.0K -box_maven-archetype_master-7AVMJNUR3VZV7RPVUBCE43H65NYTRWAOYFWSDDPCNSCWDYBAGBVQ@tmp
>>> 4.0K box_maven-ejb-plugin_master-D5MGOBWWYZCBR23TA7FG7MUJQK233O27B5TSPA7YUOAEWLP5Z72Q@tmp
>>> 4.0K -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@2@tmp
>>> 4.0K -box_maven-filtering_master-7JLC4RFGGUFPJD5SZA5D3WNJVBYMDYSH6XEWGGYSHCAMQDN7VZQA@tmp
>>> 5.7M -box_maven_pre-reset-master-AF72GWR2EBH4F7AYOLQPRDHKOEZISK7V6BEF33K5DPJKW2VE4ATA
>>> 4.0K box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@2@tmp
>>> 4.0K box_maven-rar-plugin_master-CZJVOZNWNEWP3FI6FQDKOMWKKMP7HVM2MPSDE36B4TYFH2HNQANA@tmp
>>> 4.0K -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@2@tmp
>>> 4.0K -box_maven-shared-io_master-CKUVSWZXBCH4QK62Z7UJO2FSLE5X6ZCFSHJWVQPSCR7EQORIBC7Q@tmp
>>> 4.0K box_maven-war-plugin_master-QSYAUU4F27NQH3VKQKYR34QIY6P36LH6ACXOADA2HDWCZVVU4TYA@tmp
>>> 155M    brooklyn-library-master-docker
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535791870056
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535929381454
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1535991522759
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536015781206
>>> 4.4G    brooklyn-master-build-docker_ws-cleanup_1536050912838
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536188581290
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536350369118
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536361380993
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536447781251
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536534181007
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536620581304
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536706981637
>>> 451M    brooklyn-master-build-docker_ws-cleanup_1536793381496
>>> 4.7G    brooklyn-master-build-docker_ws-cleanup_1536966181681
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1536974386104
>>> 733M    brooklyn-master-build-docker_ws-cleanup_1537055864223
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537079895664
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537087960702
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537092848128
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537116168718
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537138981521
>>> 3.1G    brooklyn-master-build-docker_ws-cleanup_1537166290489
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537252717591
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537260673390
>>> 4.8G    brooklyn-master-build-docker_ws-cleanup_1537289677014
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537425657042
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537433463888
>>> 5.0G    brooklyn-master-build-docker_ws-cleanup_1537461822531
>>> 1.4G    Calcite-Master
>>> 170M _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q
>>> 8.0K _CAMEL-12549-olingo2_update-PZQQDS6CDS5DORR2RC63COME3FNUAH4R6CMNQPYMRZXGMI47DS3Q@tmp
>>> 2.8G Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA
>>> 4.0K Camel_2.20.x-TFEL6P7XFNLJIRPYRL6K22RDL2QKGR5FX4M6GKSE64EU4BICYPVA@tmp
>>> 171M Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ
>>> 4.0K Camel_CAMEL-12622-IBFTSXAOWKDWEBFQ3GOXANPDGQSGYTS3GUJR3ZXLZJQRPUOAZYEQ@tmp
>>> 60M Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A
>>> 4.0K Camel_camel-2.20.x-SQQNE7QN3DLJQV6AZB4KZEBHA7N2TZZRYU4SJGXEFFCLXWX5UQ2A@tmp
>>> 1.9G Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ
>>> 4.0K Camel_camel-2.21.x-2HLLW2P24PAMSWCRX5CGD52T7NJ5PZVNS47EDMN4CZFYHCFKKPQQ@tmp
>>> 2.5G Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ
>>> 4.0K Camel_camel-2.22.x-BCWNGKYBULUM6XGPMTXCH5J6NJFJGFAO43BCTMPRQM6SZTDNLMYQ@tmp
>>> 3.3G    Camel.daily
>>> 4.0K    Camel.daily@tmp
>>> 2.4G Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q
>>> 4.0K Camel_master-CIGY56HHEDNGVJBWJRRHBZ64GKNJTCSYDKOPJS46W5IZTMAS3I2Q@tmp
>>> 2.3G Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ
>>> 4.0K Camel_website-YC7K6UZ6PNVBP5RVOX3M6D2U4MRT66KUPRNB45L2G7LLE63BZYVQ@tmp
>>> 810M    carbondata-master-spark-2.1
>>> 4.0K    carbondata-master-spark-2.1@tmp
>>> 584K    commons-geometry
>>> 4.0K CouchDB_master-FP4LRZTK2IWL4WCWS25JHGLJVPORIFX6ES6POVJARPVUTCZMU7SA@tmp
>>> 190M    CXF-Master-Analysis
>>> 188M    CXF-Master-JDK10
>>> 24M    CXF-Master-JDK9
>>> 1.7G    CXF-trunk-deploy
>>> 2.2G    CXF-Trunk-JDK18
>>> 74M    DeltaSpike Deploy
>>> 22M    DeltaSpike for Java EE
>>> 53M    DeltaSpike OWB 1.7.2
>>> 70M    DeltaSpike OWB 2.0.6
>>> 4.0K -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@2@tmp
>>> 4.0K -dependency-plugin_MDEP-617-ZQLYQZECSCXYYD7MV7RDO5BGII4AACSGUIH7FWUOUWHLCGKN7XPQ@tmp
>>> 7.1M    Derby-JaCoCo
>>> 7.3M    dir-ldap-api-jdk8-linux SONAR
>>> 23M    dir-ldap-api-ubuntu-deploy
>>> 2.0G    dir-studio
>>> 2.7G    dir-studio-ui-tests-in-docker
>>> 1.3M    Empire-db reports
>>> 19M    Empire-db snapshot
>>> 4.0K en-artifact-resolver_master-TCRLMCKTTCPXV2IPJG6T7K54FQQY2O7AFHHEFILBSO6ZFSIUCGRQ@tmp
>>> 4.0K en-box_maven-mapping_master-VTBNHOZQB7UZ4ORPSWLQNN55JVWPXJBJNR54WLBERZKMDQGEPMXA@tmp
>>> 4.0K en-jxr_asfMavenTlpPlgnBuild-WCSOEZUCROETKKJCNPTYSFNDV52TA7MIFL63KMUYVDEPFRT5OK4Q@tmp
>>> 4.0K en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@2@tmp
>>> 4.0K en-patch-plugin_MNGSITE-332-OEGOX4KAMT3HNPJNZR44Z57IABRBYYJTCNVPYZHJPINSHFM6IBJQ@tmp
>>> 4.0K en-toolchains-plugin_master-JSJRH7N7WHA5OITGTMSQVCI6AVNUPHTOLBYXKJXWNGGNGCYHSF5A@tmp
>>> 287M g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA
>>> 4.0K g-site_scm-project-url-list-PACBIBTTXIHT44AKXWHSV4UWSDD4I7ADSILT76TRJE5VCUTX2CTA@tmp
>>> 419M    guacamole-rest-api-test
>>> 997M HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ
>>> 4.0K HBase_Nightly_branch-1.3-7YSYBQ4PRBNZXQ26CWR7INGGD64I37BNOE6TLABRP5HTPURGHBPQ@tmp
>>> 333M HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A
>>> 4.0K HBase_Nightly_branch-1-AUHDHKRYNR4KK4DI4KZ6UHJ26DVLDRQOZOFPOYKX6D6EO3KYD56A@tmp
>>> 2.6G HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q
>>> 8.0K HBase_Nightly_branch-2.1-S5GA5UYEBNAEMXMEVWGCXQO7HNXNX47M5KZBXWNNXMFDW4LAP27Q@tmp
>>> 36M HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ
>>> 4.0K HBase_Nightly_branch-2-MO2I3RYUAQ2OT52Z2B54XSO6W34ZHP3JU2DA5UQYCAVNS747RZWQ@tmp
>>> 1.4G HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q
>>> 4.0K HBase_Nightly_HBASE-20112-STZFPQJNFOBXVX6TM3W4MPKJAH6TZRFVCCDUMAA7ASRQ6ONM4G3Q@tmp
>>> 596K HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ
>>> 4.0K HBase_Nightly_HBASE-20364-Q5VZYXODTSMLHQ4BICHKC6YELVAH3QMO5DND6NDAF3UTNDVMWXAQ@tmp
>>> 2.8G HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA
>>> 4.0K HBase_Nightly_master-4PMG3QPNOXT5YRQZS7HMZP3GLNX6XSF6DVHYXYIB5BWQ75VW3CPA@tmp
>>> 538M    Hive-trunk
>>> 3.2G    incubator-netbeans-linux
>>> 744M    incubator-netbeans-linux-experiment
>>> 4.0K -info-reports-plugin_master-32Y3P743XLZN3V3HHV4NR6ZSZMFDM7Z5PXO3RSN6SA3U6NVU4ZTA@tmp
>>> 108K    infra-test-ant-ubuntu
>>> 12K    infra-test-docker-ubuntu
>>> 60K    infra-test-jdk-ubuntu
>>> 20K    infra-test-jdk-ubuntu-harmony
>>> 40K    infra-test-jdk-ubuntu-ibm
>>> 56K    infra-test-maven-ubuntu
>>> 12K    infra-test-snap-ubuntu
>>> 12K    infra-test-xvfb
>>> 1.4M    Ivy-Snapshot-Deploy
>>> 99M    Ivy-tests-ubuntu
>>> 16M    Ivy-tests-Windows
>>> 12M    Jackrabbit-trunk
>>> 4.0K javadoc-plugin_MJAVADOC-444-JPYTCQM73CF7UEOSYS6Z2MLTMHWVHYCB5QGVIGAH4MONP6SX7OHA@tmp
>>> 4.0K javadoc-plugin_MJAVADOC-516-Z27AS7O3YE3KP4HN2X3EMP56QPDDJR4P5EBKM2B5MKLXDNLXZ3DA@tmp
>>> 19M    Jena_Development_Test
>>> 731M    Jena_JDK11
>>> 264M    kafka-1.0-jdk7
>>> 290M    kafka-1.1-jdk7
>>> 501M    kafka-2.0-jdk8
>>> 628M    kafka-pr-jdk10-scala2.12
>>> 582M    kafka-pr-jdk8-scala2.11
>>> 523M    kafka-trunk-jdk8
>>> 60M    karaf-cave-master
>>> 4.8G    karaf-master
>>> 344M    Log4j 2 3.x
>>> 173M Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ
>>> 93M Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2
>>> 4.0K Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@2@tmp
>>> 4.0K Log4jCore_master-7CCUSWJNA426OZKLRJWUDV3PDMWZHLNIFTRT7HQZXCV5SQ2476XQ@tmp
>>> 4.0K ly-plugin_FIXING-JDK-ISSUES-IEX46HBDBJ6NRUCALBPK7SVRYXYL7WLKQAOOVWNB63FAXVHXQWMQ@tmp
>>> 11M maticModuleNameImprovements-5CTZJI7MCAG23DNAIMMOTYFEI4EKBOZ2AOP3EOG34Q46K3EBQVBA
>>> 4.0K _maven-archiver_MSHARED-721-QXH4Z3WHL2IJTFJJIFNELXYZM6YOEOOXEJCOQA7JK5JMIS42PNSA@tmp
>>> 4.0K maven-box_maven-jxr_JXR-110-R2OIXR7IG7PF3QUMPP2IV53ZAKEOGWXENXIP7G2LROL7YOEOZZTA@tmp
>>> 4.0K maven-box_maven-jxr_JXR-125-KFC5JZL777UAPG4UY6F434EV6BJ2ZRPG7HBO32YUS3UJ56NWROZQ@tmp
>>> 4.0K maven-box_maven-jxr_master-HV7BHOKN2YNGQ4SOGMW6QYUNHKCZDFS3I74XW3IMRQXBAUKY6MSQ@tmp
>>> 8.0K maven-box_maven_master-DVEN5KAF4FLE3WDK7SWFNYUY4YZ7X45JVWYY6FZT5ZEB3CLWABVQ
>>> 5.7M maven-box_maven_MNG-5868-TIXVUCFHOPHJ7LSBIS4NVVVTXHOVMMQO4W2BXVF455AQYGYZ26SA
>>> 8.0K maven-box_maven_MNG-6391-BFSVDMPDHLGZMTGMVMMLWPKG4K2P6CHHPIPVW27SUVBB6UQWV7YA
>>> 8.0K maven-box_maven_MNG-6475-LMWI4CBOH2JNDHX4PPHOWPHEASG5JKLTTZW3URJSHEBTPFA2447Q
>>> 4.0K maven-box_maven-osgi_master-537V4ALFOX2ZNQAIHHNQL5TC7LIKAHUUTBK4WE42PW2CUAN3GTBQ@tmp
>>> 4.0K maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@2@tmp
>>> 4.0K maven-box_maven-scm_master-LZ7UVRJQGWEFPDXKGVXHDWRQ4JZLLSJWYQRZLEZSFZG4LONP4W2A@tmp
>>> 4.0K maven-changes-plugin_master-O3PQ7WZZHAA2T5PBIGGQZFFXFD6PLMGLZHAF4U525J6NJFJA5IMQ@tmp
>>> 4.0K maven-install-plugin_master-3R4FSR6ITADJX4LXJM433BEETYXHTGVGRU6VEEEHH2PU3W5LNT6A@tmp
>>> 4.0K _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@2@tmp
>>> 4.0K _maven-jlink-plugin_CLEANUP-YYRC5VLF7HAIAIEICXGYPF25BOQW6NXDQRTJOEZNHEUF2NVU67EA@tmp
>>> 4.0K _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@2@tmp
>>> 4.0K _maven-project-utils_master-YFJOAALTBBMYPILDHBYJYDFNTT6HPH3JYBY5B7IKICAXCAIG7IYA@tmp
>>> 4.0K maven-reporting-exec_master-XY3GEBFQ42VNXZOS5MVD7PZKI6HQ7WOMNLQJ5IUZ7XSHVEI5U6RA@tmp
>>> 5.0G    Mesos-Buildbot
>>> 1.2M    Mesos-Reviewbot
>>> 4.0K mpiler-plugin_MCOMPILER-320-LDDLDH7HQQYXD6B2RHESP35HSEBJ26NA355LSJP2PHGG3DW43OPQ@tmp
>>> 4.0K m-publish-plugin_MSCMPUB-38-ULK6I7N2ZA3FU5JGEWCSL4ODBTZDTTC77NU7CUHQJKPNE7QVYS7A@tmp
>>> 4.0K n-archiver_MSHARED-668-test-WG4DQP6EJS2N73J3VGM2JH6SMRRHF477RDRLQRWF4WAHIMWMG5OA@tmp
>>> 4.0K n-box_maven-enforcer_master-KHI64AG3D4VZKUKLMNAYHKONI5AIQBLJ5ZKYGGSW7DAZVXCJBFHQ@tmp
>>> 4.0K n-box_maven-release_cleanup-YGG2KC5W5NBVLT77VULX2CHSYMVGRKDIZ7FJM54R7TMFDTDO6JYA@tmp
>>> 4.0K n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@2@tmp
>>> 4.0K n-box_maven-surefire_master-CK52RLZV7R5PVUUYTYVCEEENMKBFPX7NV3GV6YB4YGV3EM4ZIPOQ@tmp
>>> 4.0K n-box_maven-wagon_WAGON-482-U5UQDLHLBEYZ6YKA2XFN2FQ74OK76EBRADZ4AK62MZWPJOQW2HDQ@tmp
>>> 4.0K n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@2@tmp
>>> 4.0K n-box_maven-wagon_WAGON-526-MVYPB7RAMW5VZ4RA4NSMECE376IR6WO6PB7AIWCZTHT5KJGFZFHQ@tmp
>>> 4.0K n-box_maven-wagon_WAGON-527-KWYDRBGRPK7Q2YVCYHYJDNDOSEOHGQPJHV5AHJB4GNG66RKBHCBA@tmp
>>> 28K    Nemo
>>> 4.0K nfo-reports-plugin_MPIR-366-S464FI4ATSVOULE3N7MCKV5FYYG6GVHDD7OLISMYBGUEPLQM2UNA@tmp
>>> 5.7M _not-updated-MJAR-MCOMPILER-EYMH2UKFHCTFDWW4GZGG3AX4OJLGLPGZO6TAQGLBDBFX4K6CQ5GQ
>>> 4.0K n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@2@tmp
>>> 4.0K n-resolver-ant-tasks_master-PZI36OPNRAJBXJOAPEQN5EKT3YVHZTR3KX72Y4FQGBMQE6GFTARQ@tmp
>>> 4.0K n-scm-publish-plugin_master-2GNHI3CFRHSCHWWMDIRTSWMJIE3KK6DFWPSDNLSGKYY2ZVCACIZA@tmp
>>> 4.0K n-testing_MPLUGINTESTING-55-AWMCSLLBQGJRKNRZR27EZX2B5SVGADVMVK4FGA6YEL5S4OR3TOZQ@tmp
>>> 4.0K n-testing_MPLUGINTESTING-58-UONNAARVYXAHZ4RVLUBM5ODIM6A4TFB5CEC4ELRKMGU3KDYXAPIQ@tmp
>>> 800K nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA
>>> 4.0K nts-spring-taskqueue_master-DEHCJRMAJ4RCCKE7MZYOUSHRWFQYJUMVRFF7GSAKS7JROVA4UVRA@tmp
>>> 343M    OpenWhisk-DockerHub
>>> 4.0K    OpenWhisk-DockerHub@tmp
>>> 4.0K ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@2@tmp
>>> 4.0K ox_maven-doap-plugin_master-K56SP7WQZT7DLJESN4Z67EH56OK5QEQZKI2M7VJKWLIVTWMNHWNA@tmp
>>> 4.0K ox_maven-fluido-skin_master-6RM5JATOOS6MTECVSVHV7YNCQIINRWOBDST2UCUIKNH64TYTME6Q@tmp
>>> 4.0K ox_maven-help-plugin_master-U66QJWH3TRYC4M2J55DEV6AO7VQ2JW77H5GHKBUOBKZTW32WDTTA@tmp
>>> 4.0K ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@2@tmp
>>> 4.0K ox_maven-jmod-plugin_master-35YKHMTP6DUYFXQHM4NKZCKZAUADGHEV3PUSPURFALNQQYRXE5BQ@tmp
>>> 4.0K ox_maven-site-plugin_master-LZWAKOPRMO5JAZW5HRA2IQ56LLTFU3PLKDIE2JNFFXEGASBYKK3Q@tmp
>>> 667M    PDFBox-trunk
>>> 171M    PDFBox-Trunk-jdk10
>>> 52M    Phoenix-4.x-HBase-1.2
>>> 1.6G    Phoenix-4.x-HBase-1.3
>>> 6.7M    PreCommit-Admin
>>> 2.4G pulsar_precommit_integrationtests_ws-cleanup_1537313772888
>>> 4.4G pulsar_precommit_integrationtests_ws-cleanup_1537469839992
>>> 331M    Qpid-Broker-J-7.0.x-TestMatrix
>>> 331M    Qpid-Broker-J-Artefact-Release
>>> 339M    Qpid-Broker-J-JoramJMSTest
>>> 733M    Qpid-Broker-J-Python-Test
>>> 240M    Qpid-Broker-J-Test-IBMJDK1.8
>>> 12M    Qpid-Java-Java-6.1.x-TestMatrix
>>> 29M    Qpid-proton-c
>>> 1.3M    Qpid-proton-j
>>> 12M    ranger-0.7-snapshot-publish
>>> 256M    Ratis-Master
>>> 4.0K rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@2@tmp
>>> 4.0K rces-plugin_MRRESOURCES-110-PEC2YE7NKYL6E63GN7TSRMPI5Q3DHSCF7W5XK77IXP5MYQPGBVDA@tmp
>>> 125M    samza-pr-checks
>>> 4.0K sembly-plugin_MASSEMBLY-892-AIWNXXLMVTBQ62GDGAZ5RR52AYYD2PS4MNOQBOW2K6A6AS4WHTGQ@tmp
>>> 50M    ServiceMix-Bundles
>>> 36M sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A
>>> 4.0K sling-site_master-RGGZSQNSRE77KJSAWGA5ISC4OHTBYZLT56LJVWLLL57HSX2R5R2A@tmp
>>> 282M    Struts-2-3-JDK6
>>> 4.0K -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@2@tmp
>>> 4.0K -surefire_V30_SUREFIRE-1535-6LAZP67UAOYQ7KUSYGA75JJSKWQN3XZICMKBDXXTUO5K7RMMATWQ@tmp
>>> 16M    Syncope-2_0_X-deploy
>>> 4.3G    Syncope-2_1_X-deploy
>>> 16M    Syncope-master-deploy
>>> 21M    Tamaya-Javadoc-Master
>>> 23M    Tamaya-Master-branch
>>> 74M    Tobago Trunk
>>> 81M    Tobago Trunk Sonar
>>> 4.0K    Tomcat-7.x-Maven
>>> 350M    trafficcontrol-master-build
>>> 4.0K    trafficcontrol-master-build@tmp
>>> 115M    trafficcontrol-PR-rat
>>> 41M    trafficcontrol-PR-trafficops-test
>>> 162M    trafficcontrol-traffic_ops-test
>>> 4.0K    trafficcontrol-traffic_ops-test@tmp
>>> 590M    unomi-master
>>> 4.0K ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@2@tmp
>>> 4.0K ven-archetype_ARCHETYPE-551-HRU2Q2KOTZQ5SCJRJRFH54HWHKWPNEX6PLYZFXKJEPWSWHHELAOA@tmp
>>> 4.0K ven-box_maven-surefire_1532-O6EC5ODCIKDKZL25QDLH6QFMFZD5LDHASAYEMDZSNREZV4N37X7A@tmp
>>> 4.0K ven-scripting-plugin_master-F7DFWWXZD7KAQGDPAQML3SYNOE2CUDOJDQGTJN7NKKPXY3SWUHCQ@tmp
>>> 4.0K ven-site-plugin_MNGSITE-332-4LYPVCGK77QO5G6UKI6IR7JSEN5EMK7EVKG6TK2D3VYBAHH6QPBA@tmp
>>> 4.0K x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@2@tmp
>>> 4.0K x_maven-clean-plugin_master-I6NGCLTWS42DBQD52EFLHANYNHM7TDPGSN6PDGRJS5IR6XUDW46Q@tmp
>>> 4.0K x_maven-invoker_MSHARED-752-O2LI4W4PL6ARSMCGVOOVX6GZJCOYVGXMTNU5KTGD65M6CNMTDSRQ@tmp
>>> 4.0K x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@2@tmp
>>> 4.0K x_maven-jdeps-plugin_master-NK4576UNJJCYQVC667D46I6VDU64O3SNLDEXNWG3R2K63JIOVMYA@tmp
>>> 4.0K x_maven-jlink-plugin_master-TWLBBVLGC7P5GUJSJEP6SD46MW2WP64SO5MED7OJBUOM66RIHIJQ@tmp
>>> 4.0K x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@2@tmp
>>> 4.0K x_maven-patch-plugin_master-AJNCC2MTSOLL2XNT6A33RFSXCRCGBAJM5IANLZY75KP6MNCEGTWQ@tmp
>>> 4.0K x_maven-release_IMPROVE_POM-KZP6VA3ZSGGYNTVVOCGLUBHYECJZQ34VIS6H2U5HNYVKLJFCFQKQ@tmp
>>> 4.0K x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@2@tmp
>>> 4.0K x_maven-shared-utils_master-2E2DRWROHLNJSGQVJQOXBED6MNLVPJLSLU6DGTX6YKEIJNHAYWWA@tmp
>>> 4.0K x_maven-stage-plugin_master-DUIB3VWMFIP4K7X4DPTLYSUTRVJKIKKQPIMXJ3UK6V5VUGFJHXHQ@tmp
>>> 4.0K x_maven-wagon_wagon-scm-git-JFLMMYBLUJZHZUM4KST663ZOB5W6QQB2OF5JLK4YVV3OMUVRSJDA@tmp
>>> 340M    Yetus - OWASP dependency-check cache
>>> 119M    zeppelin-pull-request
>>> 93M    ZooKeeper-Find-Flaky-Tests
>>> 465M    ZooKeeper-trunk-openjdk10
>>> 157M    ZooKeeper-trunk-owasp
>>> 119M    ZooKeeper_UT_Stress
>> 
>