You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Damian Gadomski <da...@polidea.com> on 2020/08/04 12:22:12 UTC
Re: No space left on device - beam-jenkins 1 and 7
I did a small research on the temporary directories. Seems that there's no
one unified way of telling applications to use a specific path. Neither the
guarantee that all of them will use the dedicated custom directory. Badly
behaving apps could always hardcode `/tmp`, e.g. Java ;)
But, we should be able to handle most of the cases by setting the TEMPDIR
env variable (and also less popular `TMP`, `TEMP`) and passing Java
property `java.io.tmpdir` to the builds.
There's even a plugin [1] that perfectly fits our needs. But it's
unmaintained for 3 years and not available in Jenkins plugin repository.
Not sure if we want to use it anyway. Alternatively, we can add the envs,
property, and the creation of the directory manually to the DSL scripts.
[1] https://github.com/acrolinx/tmpdir-jenkins-plugin
On Wed, Jul 29, 2020 at 12:58 AM Kenneth Knowles <ke...@apache.org> wrote:
> Cool. If it is /home/jenkins it should be just fine. Thanks for checking!
>
> Kenn
>
> On Tue, Jul 28, 2020 at 10:23 AM Damian Gadomski <
> damian.gadomski@polidea.com> wrote:
>
>> Sorry, mistake while copying, [1] should be:
>> [1]
>> https://github.com/apache/beam/blob/8aca8ccc7f1a14516ad769b63845ddd4dc163d92/.test-infra/jenkins/CommonJobProperties.groovy#L63
>>
>>
>> On Tue, Jul 28, 2020 at 7:21 PM Damian Gadomski <
>> damian.gadomski@polidea.com> wrote:
>>
>>> That's interesting. I didn't check that myself but all the Jenkins jobs
>>> are configured to wipe the workspace just before the actual build happens
>>> [1]
>>> <https://github.com/apache/beam/blob/8aca8ccc7f1a14516ad769b63845ddd4dc163d92/.test-infra/jenkins/CommonJobProperties.groovy#L6>.
>>> Git SCM plugin is used for that and it enables the option called "Wipe out
>>> repository and force clone". Docs state that it "deletes the contents of
>>> the workspace before build and before checkout" [2]
>>> <https://plugins.jenkins.io/git/>. Therefore I assume that removing
>>> workspace just after the build won't change anything.
>>>
>>> The ./.gradle/caches/modules-2/files-2.1 dir is indeed present on the
>>> worker machines but it's rather in /home/jenkins dir.
>>>
>>> damgad@apache-ci-beam-jenkins-13:/home/jenkins/.gradle$ sudo du -sh
>>> 11G .
>>> damgad@apache-ci-beam-jenkins-13:/home/jenkins/.gradle$ sudo du -sh
>>> caches/modules-2/files-2.1
>>> 2.3G caches/modules-2/files-2.1
>>>
>>> I can't find that directory structure inside workspaces.
>>>
>>> damgad@apache-ci-beam-jenkins-13:/home/jenkins/jenkins-slave/workspace$
>>> sudo find -name "files-2.1"
>>> damgad@apache-ci-beam-jenkins-13:/home/jenkins/jenkins-slave/workspace$
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/8aca8ccc7f1a14516ad769b63845ddd4dc163d92/.test-infra/jenkins/CommonJobProperties.groovy#L6
>>> [2] https://plugins.jenkins.io/git/
>>>
>>> On Tue, Jul 28, 2020 at 5:47 PM Kenneth Knowles <ke...@apache.org> wrote:
>>>
>>>> Just checking - will this wipe out dependency cache? That will slow
>>>> things down and significantly increase flakiness. If I recall correctly,
>>>> the default Jenkins layout was:
>>>>
>>>> /home/jenkins/jenkins-slave/workspace/$jobname
>>>> /home/jenkins/jenkins-slave/workspace/$jobname/.m2
>>>> /home/jenkins/jenkins-slave/workspace/$jobname/.git
>>>>
>>>> Where you can see that it did a `git clone` right into the root
>>>> workspace directory, adjacent to .m2. This was not hygienic. One important
>>>> thing was that `git clean` would wipe the maven cache with every build. So
>>>> in https://github.com/apache/beam/pull/3976 we changed it to
>>>>
>>>> /home/jenkins/jenkins-slave/workspace/$jobname
>>>> /home/jenkins/jenkins-slave/workspace/$jobname/.m2
>>>> /home/jenkins/jenkins-slave/workspace/$jobname/src/.git
>>>>
>>>> Now the .m2 directory survives and we do not constantly see flakes
>>>> re-downloading deps that are immutable. This does, of course, use disk
>>>> space.
>>>>
>>>> That was in the maven days. Gradle is the same except for $HOME/.m2 is
>>>> replaced by $HOME/.gradle/caches/modules-2/files-2.1. Is Jenkins configured
>>>> the same way so we will be wiping out the dependencies? If so, can you
>>>> address this issue? Everything in that directory should be immutable and
>>>> just a cache to avoid pointless re-download.
>>>>
>>>> Kenn
>>>>
>>>> On Tue, Jul 28, 2020 at 2:25 AM Damian Gadomski <
>>>> damian.gadomski@polidea.com> wrote:
>>>>
>>>>> Agree with Udi, workspaces seem to be the third culprit, not yet
>>>>> addressed in any way (until PR#12326
>>>>> <https://github.com/apache/beam/pull/12326> is merged). I feel that
>>>>> it'll solve the issue of filling up the disks for a long time ;)
>>>>>
>>>>> I'm also OK with moving /tmp cleanup to option B, and will happily
>>>>> investigate on proper TMPDIR config.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Jul 28, 2020 at 3:07 AM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> What about the workspaces, which can take up 175GB in some cases (see
>>>>>> above)?
>>>>>> I'm working on getting them cleaned up automatically:
>>>>>> https://github.com/apache/beam/pull/12326
>>>>>>
>>>>>> My opinion is that we would get more mileage out of fixing the jobs
>>>>>> that leave behind files in /tmp and images/containers in Docker.
>>>>>> This would also help keep development machines clean.
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 27, 2020 at 5:31 PM Tyson Hamilton <ty...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Here is a summery of how I understand things,
>>>>>>>
>>>>>>> - /tmp and /var/lib/docker are the culprit for filling up disks
>>>>>>> - inventory Jenkins job runs every 12 hours and runs a docker
>>>>>>> prune to clean up images older than 24hr
>>>>>>> - crontab on each machine cleans up /tmp files older than three
>>>>>>> days weekly
>>>>>>>
>>>>>>> This doesn't seem to be working since we're still running out of
>>>>>>> disk periodically and requiring manual intervention. Knobs and options we
>>>>>>> have available:
>>>>>>>
>>>>>>> 1. increase frequency of deleting files
>>>>>>> 2. decrease the number of days required to delete a file (e.g.
>>>>>>> older than 2 days)
>>>>>>>
>>>>>>> The execution methods we have available are:
>>>>>>>
>>>>>>> A. cron
>>>>>>> - pro: runs even if a job gets stuck in Jenkins due to full disk
>>>>>>> - con: config baked into VM which is tough to update, not
>>>>>>> discoverable or documented well
>>>>>>> B. inventory job
>>>>>>> - pro: easy to update, runs every 12h already
>>>>>>> - con: could get stuck if Jenkins agent runs out of disk or is
>>>>>>> otherwise stuck, tied to all other inventory job frequency
>>>>>>> C. configure startup scripts for the VMs that set up the cron job
>>>>>>> anytime the VM is restarted
>>>>>>> - pro: similar to A. and easy to update
>>>>>>> - con: similar to A.
>>>>>>>
>>>>>>> Between the three I prefer B. because it is consistent with other
>>>>>>> inventory jobs. If it ends up that stuck jobs prohibit scheduling of the
>>>>>>> inventory job often we could further investigate C to avoid having to
>>>>>>> rebuild the VM images repeatedly.
>>>>>>>
>>>>>>> Any objections or comments? If not, we'll go forward with B. and
>>>>>>> reduce the date check from 3 days to 2 days.
>>>>>>>
>>>>>>>
>>>>>>> On 2020/07/24 20:13:29, Ahmet Altay <al...@google.com> wrote:
>>>>>>> > Tests may not be doing docker cleanup. Inventory job runs a docker
>>>>>>> prune
>>>>>>> > every 12 hours for images older than 24 hrs [1]. Randomly looking
>>>>>>> at one of
>>>>>>> > the recent runs [2], it cleaned up a long list of containers
>>>>>>> consuming
>>>>>>> > 30+GB space. That should be just 12 hours worth of containers.
>>>>>>> >
>>>>>>> > [1]
>>>>>>> >
>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L69
>>>>>>> > [2]
>>>>>>> >
>>>>>>> https://ci-beam.apache.org/job/beam_Inventory_apache-beam-jenkins-14/501/console
>>>>>>> >
>>>>>>> > On Fri, Jul 24, 2020 at 1:07 PM Tyson Hamilton <ty...@google.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > > Yes, these are on the same volume in the /var/lib/docker
>>>>>>> directory. I'm
>>>>>>> > > unsure if they clean up leftover images.
>>>>>>> > >
>>>>>>> > > On Fri, Jul 24, 2020 at 12:52 PM Udi Meiri <eh...@google.com>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > >> I forgot Docker images:
>>>>>>> > >>
>>>>>>> > >> ehudm@apache-ci-beam-jenkins-3:~$ sudo docker system df
>>>>>>> > >> TYPE TOTAL ACTIVE SIZE
>>>>>>> > >> RECLAIMABLE
>>>>>>> > >> Images 88 9
>>>>>>> 125.4GB
>>>>>>> > >> 124.2GB (99%)
>>>>>>> > >> Containers 40 4
>>>>>>> 7.927GB
>>>>>>> > >> 7.871GB (99%)
>>>>>>> > >> Local Volumes 47 0
>>>>>>> 3.165GB
>>>>>>> > >> 3.165GB (100%)
>>>>>>> > >> Build Cache 0 0 0B
>>>>>>> > >> 0B
>>>>>>> > >>
>>>>>>> > >> There are about 90 images on that machine, with all but 1 less
>>>>>>> than 48
>>>>>>> > >> hours old.
>>>>>>> > >> I think the docker test jobs need to try harder at cleaning up
>>>>>>> their
>>>>>>> > >> leftover images. (assuming they're already doing it?)
>>>>>>> > >>
>>>>>>> > >> On Fri, Jul 24, 2020 at 12:31 PM Udi Meiri <eh...@google.com>
>>>>>>> wrote:
>>>>>>> > >>
>>>>>>> > >>> The additional slots (@3 directories) take up even more space
>>>>>>> now than
>>>>>>> > >>> before.
>>>>>>> > >>>
>>>>>>> > >>> I'm testing out https://github.com/apache/beam/pull/12326
>>>>>>> which could
>>>>>>> > >>> help by cleaning up workspaces after a run (just started a
>>>>>>> seed job).
>>>>>>> > >>>
>>>>>>> > >>> On Fri, Jul 24, 2020 at 12:13 PM Tyson Hamilton <
>>>>>>> tysonjh@google.com>
>>>>>>> > >>> wrote:
>>>>>>> > >>>
>>>>>>> > >>>> 664M beam_PreCommit_JavaPortabilityApi_Commit
>>>>>>> > >>>> 656M beam_PreCommit_JavaPortabilityApi_Commit@2
>>>>>>> > >>>> 611M beam_PreCommit_JavaPortabilityApi_Cron
>>>>>>> > >>>> 616M beam_PreCommit_JavaPortabilityApiJava11_Commit
>>>>>>> > >>>> 598M beam_PreCommit_JavaPortabilityApiJava11_Commit@2
>>>>>>> > >>>> 662M beam_PreCommit_JavaPortabilityApiJava11_Cron
>>>>>>> > >>>> 2.9G beam_PreCommit_Portable_Python_Commit
>>>>>>> > >>>> 2.9G beam_PreCommit_Portable_Python_Commit@2
>>>>>>> > >>>> 1.7G beam_PreCommit_Portable_Python_Commit@3
>>>>>>> > >>>> 3.4G beam_PreCommit_Portable_Python_Cron
>>>>>>> > >>>> 1.9G beam_PreCommit_Python2_PVR_Flink_Commit
>>>>>>> > >>>> 1.4G beam_PreCommit_Python2_PVR_Flink_Cron
>>>>>>> > >>>> 1.3G beam_PreCommit_Python2_PVR_Flink_Phrase
>>>>>>> > >>>> 6.2G beam_PreCommit_Python_Commit
>>>>>>> > >>>> 7.5G beam_PreCommit_Python_Commit@2
>>>>>>> > >>>> 7.5G beam_PreCommit_Python_Cron
>>>>>>> > >>>> 1012M beam_PreCommit_PythonDocker_Commit
>>>>>>> > >>>> 1011M beam_PreCommit_PythonDocker_Commit@2
>>>>>>> > >>>> 1011M beam_PreCommit_PythonDocker_Commit@3
>>>>>>> > >>>> 1002M beam_PreCommit_PythonDocker_Cron
>>>>>>> > >>>> 877M beam_PreCommit_PythonFormatter_Commit
>>>>>>> > >>>> 988M beam_PreCommit_PythonFormatter_Cron
>>>>>>> > >>>> 986M beam_PreCommit_PythonFormatter_Phrase
>>>>>>> > >>>> 1.7G beam_PreCommit_PythonLint_Commit
>>>>>>> > >>>> 2.1G beam_PreCommit_PythonLint_Cron
>>>>>>> > >>>> 7.5G beam_PreCommit_Python_Phrase
>>>>>>> > >>>> 346M beam_PreCommit_RAT_Commit
>>>>>>> > >>>> 341M beam_PreCommit_RAT_Cron
>>>>>>> > >>>> 338M beam_PreCommit_Spotless_Commit
>>>>>>> > >>>> 339M beam_PreCommit_Spotless_Cron
>>>>>>> > >>>> 5.5G beam_PreCommit_SQL_Commit
>>>>>>> > >>>> 5.5G beam_PreCommit_SQL_Cron
>>>>>>> > >>>> 5.5G beam_PreCommit_SQL_Java11_Commit
>>>>>>> > >>>> 750M beam_PreCommit_Website_Commit
>>>>>>> > >>>> 750M beam_PreCommit_Website_Commit@2
>>>>>>> > >>>> 750M beam_PreCommit_Website_Cron
>>>>>>> > >>>> 764M beam_PreCommit_Website_Stage_GCS_Commit
>>>>>>> > >>>> 771M beam_PreCommit_Website_Stage_GCS_Cron
>>>>>>> > >>>> 336M beam_Prober_CommunityMetrics
>>>>>>> > >>>> 693M beam_python_mongoio_load_test
>>>>>>> > >>>> 339M beam_SeedJob
>>>>>>> > >>>> 333M beam_SeedJob_Standalone
>>>>>>> > >>>> 334M beam_sonarqube_report
>>>>>>> > >>>> 556M beam_SQLBigQueryIO_Batch_Performance_Test_Java
>>>>>>> > >>>> 175G total
>>>>>>> > >>>>
>>>>>>> > >>>> On Fri, Jul 24, 2020 at 12:04 PM Tyson Hamilton <
>>>>>>> tysonjh@google.com>
>>>>>>> > >>>> wrote:
>>>>>>> > >>>>
>>>>>>> > >>>>> Ya looks like something in the workspaces is taking up room:
>>>>>>> > >>>>>
>>>>>>> > >>>>> @apache-ci-beam-jenkins-8:/home/jenkins$ sudo du -shc .
>>>>>>> > >>>>> 191G .
>>>>>>> > >>>>> 191G total
>>>>>>> > >>>>>
>>>>>>> > >>>>>
>>>>>>> > >>>>> On Fri, Jul 24, 2020 at 11:44 AM Tyson Hamilton <
>>>>>>> tysonjh@google.com>
>>>>>>> > >>>>> wrote:
>>>>>>> > >>>>>
>>>>>>> > >>>>>> Node 8 is also full. The partition that /tmp is on is here:
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>> > >>>>>> /dev/sda1 485G 482G 2.9G 100% /
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> however after cleaning up tmp with the crontab command,
>>>>>>> there is only
>>>>>>> > >>>>>> 8G usage yet it still remains 100% full:
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> @apache-ci-beam-jenkins-8:/tmp$ sudo du -shc /tmp
>>>>>>> > >>>>>> 8.0G /tmp
>>>>>>> > >>>>>> 8.0G total
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> The workspaces are in the
>>>>>>> /home/jenkins/jenkins-slave/workspace
>>>>>>> > >>>>>> directory. When I run a du on that, it takes really long.
>>>>>>> I'll let it keep
>>>>>>> > >>>>>> running for a while to see if it ever returns a result but
>>>>>>> so far this
>>>>>>> > >>>>>> seems suspect.
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>
>>>>>>> > >>>>>> On Fri, Jul 24, 2020 at 11:19 AM Tyson Hamilton <
>>>>>>> tysonjh@google.com>
>>>>>>> > >>>>>> wrote:
>>>>>>> > >>>>>>
>>>>>>> > >>>>>>> Everything I've been looking at is in the /tmp dir. Where
>>>>>>> are the
>>>>>>> > >>>>>>> workspaces, or what are the named?
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>> On Fri, Jul 24, 2020 at 11:03 AM Udi Meiri <
>>>>>>> ehudm@google.com> wrote:
>>>>>>> > >>>>>>>
>>>>>>> > >>>>>>>> I'm curious to what you find. Was it /tmp or the
>>>>>>> workspaces using
>>>>>>> > >>>>>>>> up the space?
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>> On Fri, Jul 24, 2020 at 10:57 AM Tyson Hamilton <
>>>>>>> tysonjh@google.com>
>>>>>>> > >>>>>>>> wrote:
>>>>>>> > >>>>>>>>
>>>>>>> > >>>>>>>>> Bleck. I just realized that it is 'offline' so that
>>>>>>> won't work.
>>>>>>> > >>>>>>>>> I'll clean up manually on the machine using the cron
>>>>>>> command.
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>> On Fri, Jul 24, 2020 at 10:56 AM Tyson Hamilton <
>>>>>>> > >>>>>>>>> tysonjh@google.com> wrote:
>>>>>>> > >>>>>>>>>
>>>>>>> > >>>>>>>>>> Something isn't working with the current set up because
>>>>>>> node 15
>>>>>>> > >>>>>>>>>> appears to be out of space and is currently 'offline'
>>>>>>> according to Jenkins.
>>>>>>> > >>>>>>>>>> Can someone run the cleanup job? The machine is full,
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> @apache-ci-beam-jenkins-15:/tmp$ df -h
>>>>>>> > >>>>>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>> > >>>>>>>>>> udev 52G 0 52G 0% /dev
>>>>>>> > >>>>>>>>>> tmpfs 11G 265M 10G 3% /run
>>>>>>> > >>>>>>>>>> */dev/sda1 485G 484G 880M 100% /*
>>>>>>> > >>>>>>>>>> tmpfs 52G 0 52G 0% /dev/shm
>>>>>>> > >>>>>>>>>> tmpfs 5.0M 0 5.0M 0% /run/lock
>>>>>>> > >>>>>>>>>> tmpfs 52G 0 52G 0% /sys/fs/cgroup
>>>>>>> > >>>>>>>>>> tmpfs 11G 0 11G 0% /run/user/1017
>>>>>>> > >>>>>>>>>> tmpfs 11G 0 11G 0% /run/user/1037
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> apache-ci-beam-jenkins-15:/tmp$ sudo du -ah --time . |
>>>>>>> sort -rhk
>>>>>>> > >>>>>>>>>> 1,1 | head -n 20
>>>>>>> > >>>>>>>>>> 20G 2020-07-24 17:52 .
>>>>>>> > >>>>>>>>>> 580M 2020-07-22 17:31
>>>>>>> ./junit1031982597110125586
>>>>>>> > >>>>>>>>>> 517M 2020-07-22 17:31
>>>>>>> > >>>>>>>>>>
>>>>>>> ./junit1031982597110125586/junit8739924829337821410/heap_dump.hprof
>>>>>>> > >>>>>>>>>> 517M 2020-07-22 17:31
>>>>>>> > >>>>>>>>>> ./junit1031982597110125586/junit8739924829337821410
>>>>>>> > >>>>>>>>>> 263M 2020-07-22 12:23 ./pip-install-2GUhO_
>>>>>>> > >>>>>>>>>> 263M 2020-07-20 09:30 ./pip-install-sxgwqr
>>>>>>> > >>>>>>>>>> 263M 2020-07-17 13:56 ./pip-install-bWSKIV
>>>>>>> > >>>>>>>>>> 242M 2020-07-21 20:25
>>>>>>> ./beam-pipeline-tempmByU6T
>>>>>>> > >>>>>>>>>> 242M 2020-07-21 20:21
>>>>>>> ./beam-pipeline-tempV85xeK
>>>>>>> > >>>>>>>>>> 242M 2020-07-21 20:15
>>>>>>> ./beam-pipeline-temp7dJROJ
>>>>>>> > >>>>>>>>>> 236M 2020-07-21 20:25
>>>>>>> > >>>>>>>>>> ./beam-pipeline-tempmByU6T/tmpOWj3Yr
>>>>>>> > >>>>>>>>>> 236M 2020-07-21 20:21
>>>>>>> > >>>>>>>>>> ./beam-pipeline-tempV85xeK/tmppbQHB3
>>>>>>> > >>>>>>>>>> 236M 2020-07-21 20:15
>>>>>>> > >>>>>>>>>> ./beam-pipeline-temp7dJROJ/tmpgOXPKW
>>>>>>> > >>>>>>>>>> 111M 2020-07-23 00:57 ./pip-install-1JnyNE
>>>>>>> > >>>>>>>>>> 105M 2020-07-23 00:17
>>>>>>> ./beam-artifact1374651823280819755
>>>>>>> > >>>>>>>>>> 105M 2020-07-23 00:16
>>>>>>> ./beam-artifact5050755582921936972
>>>>>>> > >>>>>>>>>> 105M 2020-07-23 00:16
>>>>>>> ./beam-artifact1834064452502646289
>>>>>>> > >>>>>>>>>> 105M 2020-07-23 00:15
>>>>>>> ./beam-artifact682561790267074916
>>>>>>> > >>>>>>>>>> 105M 2020-07-23 00:15
>>>>>>> ./beam-artifact4691304965824489394
>>>>>>> > >>>>>>>>>> 105M 2020-07-23 00:14
>>>>>>> ./beam-artifact4050383819822604421
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>> On Wed, Jul 22, 2020 at 12:03 PM Robert Bradshaw <
>>>>>>> > >>>>>>>>>> robertwb@google.com> wrote:
>>>>>>> > >>>>>>>>>>
>>>>>>> > >>>>>>>>>>> On Wed, Jul 22, 2020 at 11:57 AM Tyson Hamilton <
>>>>>>> > >>>>>>>>>>> tysonjh@google.com> wrote:
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> Ah I see, thanks Kenn. I found some advice from the
>>>>>>> Apache
>>>>>>> > >>>>>>>>>>>> infra wiki that also suggests using a tmpdir inside
>>>>>>> the workspace [1]:
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> Procedures Projects can take to clean up disk space
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> Projects can help themselves and Infra by taking some
>>>>>>> basic
>>>>>>> > >>>>>>>>>>>> steps to help clean up their jobs after themselves on
>>>>>>> the build nodes.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> 1. Use a ./tmp dir in your jobs workspace. That
>>>>>>> way it gets
>>>>>>> > >>>>>>>>>>>> cleaned up when job workspaces expire.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>> Tests should be (able to be) written to use the
>>>>>>> standard
>>>>>>> > >>>>>>>>>>> temporary file mechanisms, and the environment set up
>>>>>>> on Jenkins such that
>>>>>>> > >>>>>>>>>>> that falls into the respective workspaces. Ideally
>>>>>>> this should be as simple
>>>>>>> > >>>>>>>>>>> as setting the TMPDIR (or similar) environment
>>>>>>> variable (and making sure it
>>>>>>> > >>>>>>>>>>> exists/is writable).
>>>>>>> > >>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> 1. Configure your jobs to wipe workspaces on start
>>>>>>> or
>>>>>>> > >>>>>>>>>>>> finish.
>>>>>>> > >>>>>>>>>>>> 2. Configure your jobs to only keep 5 or 10
>>>>>>> previous builds.
>>>>>>> > >>>>>>>>>>>> 3. Configure your jobs to only keep 5 or 10
>>>>>>> previous
>>>>>>> > >>>>>>>>>>>> artifacts.
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> [1]:
>>>>>>> > >>>>>>>>>>>>
>>>>>>> https://cwiki.apache.org/confluence/display/INFRA/Disk+Space+cleanup+of+Jenkins+nodes
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>> On Wed, Jul 22, 2020 at 8:06 AM Kenneth Knowles <
>>>>>>> > >>>>>>>>>>>> kenn@apache.org> wrote:
>>>>>>> > >>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>> Those file listings look like the result of using
>>>>>>> standard
>>>>>>> > >>>>>>>>>>>>> temp file APIs but with TMPDIR set to /tmp.
>>>>>>> > >>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>> On Mon, Jul 20, 2020 at 7:55 PM Tyson Hamilton <
>>>>>>> > >>>>>>>>>>>>> tysonjh@google.com> wrote:
>>>>>>> > >>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> Jobs are hermetic as far as I can tell and use
>>>>>>> unique
>>>>>>> > >>>>>>>>>>>>>> subdirectories inside of /tmp. Here is a quick look
>>>>>>> into two examples:
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> @apache-ci-beam-jenkins-4:/tmp$ sudo du -ah --time
>>>>>>> . | sort
>>>>>>> > >>>>>>>>>>>>>> -rhk 1,1 | head -n 20
>>>>>>> > >>>>>>>>>>>>>> 1.6G 2020-07-21 02:25 .
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:48
>>>>>>> ./beam-pipeline-temp3ybuY4
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:46
>>>>>>> ./beam-pipeline-tempuxjiPT
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:44
>>>>>>> ./beam-pipeline-tempVpg1ME
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:42
>>>>>>> ./beam-pipeline-tempJ4EpyB
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:39
>>>>>>> ./beam-pipeline-tempepea7Q
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-17 18:35
>>>>>>> ./beam-pipeline-temp79qot2
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:48
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmpy_Ytzz
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:46
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpN5_UfJ
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:44
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpxSm8pX
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:42
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpMZJU76
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:39
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmpWy1vWX
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-17 18:35
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmpvN7vWA
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:48
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp3ybuY4/tmprlh_di
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:46
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempuxjiPT/tmpLmVWfe
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:44
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempVpg1ME/tmpvrxbY7
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:42
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempJ4EpyB/tmpLTb6Mj
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:39
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempepea7Q/tmptYF1v1
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-17 18:35
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-temp79qot2/tmplfV0Rg
>>>>>>> > >>>>>>>>>>>>>> 2.7M 2020-07-17 20:10
>>>>>>> ./pip-install-q9l227ef
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> @apache-ci-beam-jenkins-11:/tmp$ sudo du -ah --time
>>>>>>> . | sort
>>>>>>> > >>>>>>>>>>>>>> -rhk 1,1 | head -n 20
>>>>>>> > >>>>>>>>>>>>>> 817M 2020-07-21 02:26 .
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-19 12:14
>>>>>>> ./beam-pipeline-tempUTXqlM
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-19 12:11
>>>>>>> ./beam-pipeline-tempx3Yno3
>>>>>>> > >>>>>>>>>>>>>> 242M 2020-07-19 12:05
>>>>>>> ./beam-pipeline-tempyCrMYq
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-19 12:14
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpstXoL0
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-19 12:11
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpnnVn65
>>>>>>> > >>>>>>>>>>>>>> 236M 2020-07-19 12:05
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpRF0iNs
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-19 12:14
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpbJjUAQ
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-19 12:11
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmpsmmzqe
>>>>>>> > >>>>>>>>>>>>>> 3.7M 2020-07-19 12:05
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmp5b3ZvY
>>>>>>> > >>>>>>>>>>>>>> 2.0M 2020-07-19 12:14
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmpoj3orz
>>>>>>> > >>>>>>>>>>>>>> 2.0M 2020-07-19 12:11
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmptng9sZ
>>>>>>> > >>>>>>>>>>>>>> 2.0M 2020-07-19 12:05
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpWp6njc
>>>>>>> > >>>>>>>>>>>>>> 1.2M 2020-07-19 12:14
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempUTXqlM/tmphgdj35
>>>>>>> > >>>>>>>>>>>>>> 1.2M 2020-07-19 12:11
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempx3Yno3/tmp8ySXpm
>>>>>>> > >>>>>>>>>>>>>> 1.2M 2020-07-19 12:05
>>>>>>> > >>>>>>>>>>>>>> ./beam-pipeline-tempyCrMYq/tmpNVEJ4e
>>>>>>> > >>>>>>>>>>>>>> 992K 2020-07-12 12:00
>>>>>>> ./junit642086915811430564
>>>>>>> > >>>>>>>>>>>>>> 988K 2020-07-12 12:00
>>>>>>> ./junit642086915811430564/beam
>>>>>>> > >>>>>>>>>>>>>> 984K 2020-07-12 12:00
>>>>>>> > >>>>>>>>>>>>>> ./junit642086915811430564/beam/nodes
>>>>>>> > >>>>>>>>>>>>>> 980K 2020-07-12 12:00
>>>>>>> > >>>>>>>>>>>>>> ./junit642086915811430564/beam/nodes/0
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 6:46 PM Udi Meiri <
>>>>>>> ehudm@google.com>
>>>>>>> > >>>>>>>>>>>>>> wrote:
>>>>>>> > >>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>> You're right, job workspaces should be hermetic.
>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 1:24 PM Kenneth Knowles <
>>>>>>> > >>>>>>>>>>>>>>> kenn@apache.org> wrote:
>>>>>>> > >>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>> I'm probably late to this discussion and missing
>>>>>>> something,
>>>>>>> > >>>>>>>>>>>>>>>> but why are we writing to /tmp at all? I would
>>>>>>> expect TMPDIR to point
>>>>>>> > >>>>>>>>>>>>>>>> somewhere inside the job directory that will be
>>>>>>> wiped by Jenkins, and I
>>>>>>> > >>>>>>>>>>>>>>>> would expect code to always create temp files via
>>>>>>> APIs that respect this.
>>>>>>> > >>>>>>>>>>>>>>>> Is Jenkins not cleaning up? Do we not have the
>>>>>>> ability to set this up? Do
>>>>>>> > >>>>>>>>>>>>>>>> we have bugs in our code (that we could probably
>>>>>>> find by setting TMPDIR to
>>>>>>> > >>>>>>>>>>>>>>>> somewhere not-/tmp and running the tests without
>>>>>>> write permission to /tmp,
>>>>>>> > >>>>>>>>>>>>>>>> etc)
>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>> Kenn
>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 11:39 AM Ahmet Altay <
>>>>>>> > >>>>>>>>>>>>>>>> altay@google.com> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>> Related to workspace directory growth, +Udi Meiri
>>>>>>> > >>>>>>>>>>>>>>>>> <eh...@google.com> filed a relevant issue
>>>>>>> previously (
>>>>>>> > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-9865)
>>>>>>> for
>>>>>>> > >>>>>>>>>>>>>>>>> cleaning up workspace directory after successful
>>>>>>> jobs. Alternatively, we
>>>>>>> > >>>>>>>>>>>>>>>>> can consider periodically cleaning up the /src
>>>>>>> directories.
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>> I would suggest moving the cron task from
>>>>>>> internal cron
>>>>>>> > >>>>>>>>>>>>>>>>> scripts to the inventory job (
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_Inventory.groovy#L51
>>>>>>> ).
>>>>>>> > >>>>>>>>>>>>>>>>> That way, we can see all the cron jobs as part
>>>>>>> of the source tree, adjust
>>>>>>> > >>>>>>>>>>>>>>>>> frequencies and clean up codes with PRs. I do
>>>>>>> not know how internal cron
>>>>>>> > >>>>>>>>>>>>>>>>> scripts are created, maintained, and how would
>>>>>>> they be recreated for new
>>>>>>> > >>>>>>>>>>>>>>>>> worker instances.
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>> /cc +Tyson Hamilton <ty...@google.com>
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 4:50 AM Damian Gadomski <
>>>>>>> > >>>>>>>>>>>>>>>>> damian.gadomski@polidea.com> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> Hey,
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> I've recently created a solution for the
>>>>>>> growing /tmp
>>>>>>> > >>>>>>>>>>>>>>>>>> directory. Part of it is the job mentioned by
>>>>>>> Tyson:
>>>>>>> > >>>>>>>>>>>>>>>>>> *beam_Clean_tmp_directory*. It's intentionally
>>>>>>> not
>>>>>>> > >>>>>>>>>>>>>>>>>> triggered by cron and should be a last resort
>>>>>>> solution for some strange
>>>>>>> > >>>>>>>>>>>>>>>>>> cases.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> Along with that job, I've also updated every
>>>>>>> worker with
>>>>>>> > >>>>>>>>>>>>>>>>>> an internal cron script. It's being executed
>>>>>>> once a week and deletes all
>>>>>>> > >>>>>>>>>>>>>>>>>> the files (and only files) that were not
>>>>>>> accessed for at least three days.
>>>>>>> > >>>>>>>>>>>>>>>>>> That's designed to be as safe as possible for
>>>>>>> the running jobs on the
>>>>>>> > >>>>>>>>>>>>>>>>>> worker (not to delete the files that are still
>>>>>>> in use), and also to be
>>>>>>> > >>>>>>>>>>>>>>>>>> insensitive to the current workload on the
>>>>>>> machine. The cleanup will always
>>>>>>> > >>>>>>>>>>>>>>>>>> happen, even if some long-running/stuck jobs
>>>>>>> are blocking the machine.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> I also think that currently the "No space left"
>>>>>>> errors
>>>>>>> > >>>>>>>>>>>>>>>>>> may be a consequence of growing workspace
>>>>>>> directory rather than /tmp. I
>>>>>>> > >>>>>>>>>>>>>>>>>> didn't do any detailed analysis but e.g.
>>>>>>> currently, on
>>>>>>> > >>>>>>>>>>>>>>>>>> apache-beam-jenkins-7 the workspace directory
>>>>>>> size is 158 GB while /tmp is
>>>>>>> > >>>>>>>>>>>>>>>>>> only 16 GB. We should either guarantee the disk
>>>>>>> size to hold workspaces for
>>>>>>> > >>>>>>>>>>>>>>>>>> all jobs (because eventually, every worker will
>>>>>>> execute each job) or clear
>>>>>>> > >>>>>>>>>>>>>>>>>> also the workspaces in some way.
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> Regards,
>>>>>>> > >>>>>>>>>>>>>>>>>> Damian
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>> On Mon, Jul 20, 2020 at 10:43 AM Maximilian
>>>>>>> Michels <
>>>>>>> > >>>>>>>>>>>>>>>>>> mxm@apache.org> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> +1 for scheduling it via a cron job if it
>>>>>>> won't lead to
>>>>>>> > >>>>>>>>>>>>>>>>>>> test failures
>>>>>>> > >>>>>>>>>>>>>>>>>>> while running. Not a Jenkins expert but maybe
>>>>>>> there is
>>>>>>> > >>>>>>>>>>>>>>>>>>> the notion of
>>>>>>> > >>>>>>>>>>>>>>>>>>> running exclusively while no other tasks are
>>>>>>> running?
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> -Max
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> On 17.07.20 21:49, Tyson Hamilton wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>>> > FYI there was a job introduced to do this in
>>>>>>> Jenkins:
>>>>>>> > >>>>>>>>>>>>>>>>>>> beam_Clean_tmp_directory
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> > Currently it needs to be run manually. I'm
>>>>>>> seeing some
>>>>>>> > >>>>>>>>>>>>>>>>>>> out of disk related errors in precommit tests
>>>>>>> currently, perhaps we should
>>>>>>> > >>>>>>>>>>>>>>>>>>> schedule this job with cron?
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> > On 2020/03/11 19:31:13, Heejong Lee <
>>>>>>> > >>>>>>>>>>>>>>>>>>> heejong@google.com> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>>> >> Still seeing no space left on device errors
>>>>>>> on
>>>>>>> > >>>>>>>>>>>>>>>>>>> jenkins-7 (for example:
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/job/beam_PreCommit_PythonLint_Commit/2754/
>>>>>>> > >>>>>>>>>>>>>>>>>>> )
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >> On Fri, Mar 6, 2020 at 7:11 PM Alan Myrvold
>>>>>>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>> amyrvold@google.com> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> Did a one time cleanup of tmp files owned
>>>>>>> by jenkins
>>>>>>> > >>>>>>>>>>>>>>>>>>> older than 3 days.
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> Agree that we need a longer term solution.
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> Passing recent tests on all executors
>>>>>>> except
>>>>>>> > >>>>>>>>>>>>>>>>>>> jenkins-12, which has not
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> scheduled recent builds for the past 13
>>>>>>> days. Not
>>>>>>> > >>>>>>>>>>>>>>>>>>> scheduling:
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-12/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-12/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> Recent passing builds:
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-1/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-1/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-2/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-2/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-3/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-3/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-4/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-4/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-5/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-5/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-6/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-6/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-7/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-7/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-8/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-8/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-9/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-9/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-10/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-10/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-11/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-11/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-13/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-13/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-14/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-14/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-15/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-15/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://builds.apache.org/computer/apache-beam-jenkins-16/builds
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> <
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> https://www.google.com/url?q=https://builds.apache.org/computer/apache-beam-jenkins-16/builds&sa=D
>>>>>>> > >>>>>>>>>>>>>>>>>>> >
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>> On Fri, Mar 6, 2020 at 11:54 AM Ahmet
>>>>>>> Altay <
>>>>>>> > >>>>>>>>>>>>>>>>>>> altay@google.com> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>> +Alan Myrvold <am...@google.com> is
>>>>>>> doing a one
>>>>>>> > >>>>>>>>>>>>>>>>>>> time cleanup. I agree
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>> that we need to have a solution to
>>>>>>> automate this
>>>>>>> > >>>>>>>>>>>>>>>>>>> task or address the root
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>> cause of the buildup.
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>> On Thu, Mar 5, 2020 at 2:47 AM Michał
>>>>>>> Walenia <
>>>>>>> > >>>>>>>>>>>>>>>>>>> michal.walenia@polidea.com>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>> wrote:
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Hi there,
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> it seems we have a problem with Jenkins
>>>>>>> workers
>>>>>>> > >>>>>>>>>>>>>>>>>>> again. Nodes 1 and 7
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> both fail jobs with "No space left on
>>>>>>> device".
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Who is the best person to contact in
>>>>>>> these cases
>>>>>>> > >>>>>>>>>>>>>>>>>>> (someone with access
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> permissions to the workers).
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> I also noticed that such errors are
>>>>>>> becoming more
>>>>>>> > >>>>>>>>>>>>>>>>>>> and more frequent
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> recently and I'd like to discuss how can
>>>>>>> this be
>>>>>>> > >>>>>>>>>>>>>>>>>>> remedied. Can a cleanup
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> task be automated on Jenkins somehow?
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Regards
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Michal
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> --
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Michał Walenia
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Polidea <https://www.polidea.com/> |
>>>>>>> Software
>>>>>>> > >>>>>>>>>>>>>>>>>>> Engineer
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> M: +48 791 432 002
>>>>>>> <+48%20791%20432%20002> <+48%20791%20432%20002> <
>>>>>>> > >>>>>>>>>>>>>>>>>>> +48791432002 <+48%20791%20432%20002>
>>>>>>> <+48%20791%20432%20002>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> E: michal.walenia@polidea.com
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Unique Tech
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>> Check out our projects! <
>>>>>>> > >>>>>>>>>>>>>>>>>>> https://www.polidea.com/our-work>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>> >>
>>>>>>> > >>>>>>>>>>>>>>>>>>>
>>>>>>> > >>>>>>>>>>>>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>