You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Udi Meiri <eh...@google.com> on 2019/06/27 17:16:29 UTC

apache-beam-jenkins-15 out of disk

Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648

Can someone investigate what's going on?

Re: apache-beam-jenkins-15 out of disk

Posted by Yifan Zou <yi...@google.com>.
I reimaged the beam15. The worker is re-enabled. Let us know if anything
weird happens on any agent.

Thanks.
Yifan

On Mon, Jul 1, 2019 at 10:00 AM Yifan Zou <yi...@google.com> wrote:

> https://issues.apache.org/jira/browse/BEAM-7650 tracks the docker issue.
>
> On Sun, Jun 30, 2019 at 2:35 PM Mark Liu <ma...@google.com> wrote:
>
>> Thank you for triaging and working out a solution Yifan and Ankur.
>>
>> Ankur, from what you discovered, we should fix this race condition
>> otherwise same problem will happen in the future. Is there a jira tracking
>> this issue?
>>
>> On Fri, Jun 28, 2019 at 4:56 PM Yifan Zou <yi...@google.com> wrote:
>>
>>> Sorry for the inconvenience. I disabled the worker. I'll need more time
>>> to restore it.
>>>
>>> On Fri, Jun 28, 2019 at 3:56 PM Daniel Oliveira <da...@google.com>
>>> wrote:
>>>
>>>> Any updates to this issue today? It seems like this (or a similar bug)
>>>> is still happening across many Pre and Postcommits.
>>>>
>>>> On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou <yi...@google.com> wrote:
>>>>
>>>>> I did the prune on beam15. The disk was free but all jobs fails with
>>>>> other weird problems. Looks like docker prune overkills, but I don't have
>>>>> evidence. Will look further in AM.
>>>>>
>>>>> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> See how the hdfs IT already avoids tag collisions.
>>>>>>
>>>>>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang <zy...@google.com> wrote:
>>>>>>
>>>>>>> for flakiness I guess a tag is needed to separate concurrent build
>>>>>>> apart.
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>>>>>
>>>>>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> This highlights the race condition caused by using single docker
>>>>>>>>> registry on a machine.
>>>>>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>>>>>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>>>>>>>>
>>>>>>>>> Is their a way to dynamically create and destroy docker repository
>>>>>>>>> on a machine and clean all the relevant data?
>>>>>>>>>
>>>>>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The problem was because of the large quantity of stale docker
>>>>>>>>>> images generated by the Python portable tests and HDFS IT.
>>>>>>>>>>
>>>>>>>>>> Dumping the docker disk usage gives me:
>>>>>>>>>>
>>>>>>>>>> TYPE                TOTAL               ACTIVE              SIZE
>>>>>>>>>>                RECLAIMABLE
>>>>>>>>>> *Images              1039                356
>>>>>>>>>> 424GB               384.2GB (90%)*
>>>>>>>>>> Containers          987                 2
>>>>>>>>>> 2.042GB             2.041GB (99%)
>>>>>>>>>> Local Volumes       126                 0
>>>>>>>>>> 392.8MB             392.8MB (100%)
>>>>>>>>>>
>>>>>>>>>> REPOSITORY
>>>>>>>>>>                     TAG                 IMAGE ID            CREATED
>>>>>>>>>>     SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>>>>>      latest              ff1b949f4442        22 hours ago        1.639GB
>>>>>>>>>>           922.3MB                  716.9MB             0
>>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>>>>>        latest              1dda7b9d9748        22 hours ago        1.624GB
>>>>>>>>>>             913.7MB               710.3MB             0
>>>>>>>>>> <none>
>>>>>>>>>>                            <none>              05458187a0e3        22 hours
>>>>>>>>>> ago        732.9MB             625.1MB            107.8MB             4
>>>>>>>>>> <none>
>>>>>>>>>>                            <none>              896f35dd685f        23 hours
>>>>>>>>>> ago        1.639GB             922.3MB               716.9MB             0
>>>>>>>>>> <none>
>>>>>>>>>>                            <none>              db4d24ca9f2b        23 hours
>>>>>>>>>> ago        1.624GB             913.7MB              710.3MB             0
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              547df4d71c31        23
>>>>>>>>>> hours ago        732.9MB             625.1MB             107.8MB
>>>>>>>>>>   4
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              dd7d9582c3e0        23
>>>>>>>>>> hours ago        1.639GB             922.3MB             716.9MB
>>>>>>>>>>   0
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              664aae255239        23
>>>>>>>>>> hours ago        1.624GB             913.7MB             710.3MB
>>>>>>>>>>   0
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              b528fedf9228        23
>>>>>>>>>> hours ago        732.9MB             625.1MB             107.8MB
>>>>>>>>>>   4
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              8e996f22435e        25
>>>>>>>>>> hours ago        1.624GB             913.7MB            710.3MB
>>>>>>>>>> 0
>>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test
>>>>>>>>>> latest              24b73b3fec06        25 hours ago        1.305GB
>>>>>>>>>>     965.7MB               339.5MB             0
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              096325fb48de       25 hours
>>>>>>>>>> ago        732.9MB             625.1MB            107.8MB              2
>>>>>>>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>>>>>>>          latest              c36d8ff2945d          25 hours ago
>>>>>>>>>>  685.6MB             625.1MB               60.52MB             0
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              11c86ebe025f        26
>>>>>>>>>> hours ago        1.639GB             922.3MB              716.9MB
>>>>>>>>>>   0
>>>>>>>>>> <none>
>>>>>>>>>>                             <none>              2ecd69c89ec1        26
>>>>>>>>>> hours ago        1.624GB             913.7MB             710.3MB
>>>>>>>>>>   0
>>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test
>>>>>>>>>>  latest              3d1d589d44fe        2 days ago          1.305GB
>>>>>>>>>>       965.7MB               339.5MB             0
>>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test
>>>>>>>>>>  latest              d1cc503ebe8e        2 days ago          1.305GB
>>>>>>>>>>       965.7MB             339.2MB             0
>>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test
>>>>>>>>>>  latest              8582c6ca6e15        3 days ago          1.305GB
>>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test
>>>>>>>>>>  latest              4591e0948170        3 days ago          1.305GB
>>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test
>>>>>>>>>>  latest              ab181c49d56e        4 days ago          1.305GB
>>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test
>>>>>>>>>>  latest              2104ba0a6db7        4 days ago          1.305GB
>>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>>> ...
>>>>>>>>>> <1000+ images>
>>>>>>>>>>
>>>>>>>>>> I removed unused the images and the beam15 is back now.
>>>>>>>>>>
>>>>>>>>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>>>>>>>>> Ankur, I assigned the issue to you. Feel free to reassign it if
>>>>>>>>>> needed.
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>> Yifan
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Something were eating the disk. Disconnected the worker so jobs
>>>>>>>>>>> could be allocated to other nodes. Will look deeper.
>>>>>>>>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>>>>>>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I'm on it.
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Opened a bug here:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can someone investigate what's going on?
>>>>>>>>>>>>>
>>>>>>>>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Yifan Zou <yi...@google.com>.
https://issues.apache.org/jira/browse/BEAM-7650 tracks the docker issue.

On Sun, Jun 30, 2019 at 2:35 PM Mark Liu <ma...@google.com> wrote:

> Thank you for triaging and working out a solution Yifan and Ankur.
>
> Ankur, from what you discovered, we should fix this race condition
> otherwise same problem will happen in the future. Is there a jira tracking
> this issue?
>
> On Fri, Jun 28, 2019 at 4:56 PM Yifan Zou <yi...@google.com> wrote:
>
>> Sorry for the inconvenience. I disabled the worker. I'll need more time
>> to restore it.
>>
>> On Fri, Jun 28, 2019 at 3:56 PM Daniel Oliveira <da...@google.com>
>> wrote:
>>
>>> Any updates to this issue today? It seems like this (or a similar bug)
>>> is still happening across many Pre and Postcommits.
>>>
>>> On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou <yi...@google.com> wrote:
>>>
>>>> I did the prune on beam15. The disk was free but all jobs fails with
>>>> other weird problems. Looks like docker prune overkills, but I don't have
>>>> evidence. Will look further in AM.
>>>>
>>>> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> See how the hdfs IT already avoids tag collisions.
>>>>>
>>>>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang <zy...@google.com> wrote:
>>>>>
>>>>>> for flakiness I guess a tag is needed to separate concurrent build
>>>>>> apart.
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> This highlights the race condition caused by using single docker
>>>>>>>> registry on a machine.
>>>>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>>>>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>>>>>>>
>>>>>>>> Is their a way to dynamically create and destroy docker repository
>>>>>>>> on a machine and clean all the relevant data?
>>>>>>>>
>>>>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The problem was because of the large quantity of stale docker
>>>>>>>>> images generated by the Python portable tests and HDFS IT.
>>>>>>>>>
>>>>>>>>> Dumping the docker disk usage gives me:
>>>>>>>>>
>>>>>>>>> TYPE                TOTAL               ACTIVE              SIZE
>>>>>>>>>              RECLAIMABLE
>>>>>>>>> *Images              1039                356                 424GB
>>>>>>>>>               384.2GB (90%)*
>>>>>>>>> Containers          987                 2
>>>>>>>>> 2.042GB             2.041GB (99%)
>>>>>>>>> Local Volumes       126                 0
>>>>>>>>> 392.8MB             392.8MB (100%)
>>>>>>>>>
>>>>>>>>> REPOSITORY
>>>>>>>>>                     TAG                 IMAGE ID            CREATED
>>>>>>>>>     SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>>>>      latest              ff1b949f4442        22 hours ago        1.639GB
>>>>>>>>>           922.3MB                  716.9MB             0
>>>>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>>>>      latest              1dda7b9d9748        22 hours ago        1.624GB
>>>>>>>>>           913.7MB               710.3MB             0
>>>>>>>>> <none>
>>>>>>>>>                            <none>              05458187a0e3        22 hours
>>>>>>>>> ago        732.9MB             625.1MB            107.8MB             4
>>>>>>>>> <none>
>>>>>>>>>                            <none>              896f35dd685f        23 hours
>>>>>>>>> ago        1.639GB             922.3MB               716.9MB             0
>>>>>>>>> <none>
>>>>>>>>>                            <none>              db4d24ca9f2b        23 hours
>>>>>>>>> ago        1.624GB             913.7MB              710.3MB             0
>>>>>>>>> <none>
>>>>>>>>>                           <none>              547df4d71c31        23 hours
>>>>>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>>>>>> <none>
>>>>>>>>>                           <none>              dd7d9582c3e0        23 hours
>>>>>>>>> ago        1.639GB             922.3MB             716.9MB             0
>>>>>>>>> <none>
>>>>>>>>>                           <none>              664aae255239        23 hours
>>>>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>>>>> <none>
>>>>>>>>>                             <none>              b528fedf9228        23
>>>>>>>>> hours ago        732.9MB             625.1MB             107.8MB
>>>>>>>>>   4
>>>>>>>>> <none>
>>>>>>>>>                             <none>              8e996f22435e        25
>>>>>>>>> hours ago        1.624GB             913.7MB            710.3MB
>>>>>>>>> 0
>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test
>>>>>>>>> latest              24b73b3fec06        25 hours ago        1.305GB
>>>>>>>>>     965.7MB               339.5MB             0
>>>>>>>>> <none>
>>>>>>>>>                             <none>              096325fb48de       25 hours
>>>>>>>>> ago        732.9MB             625.1MB            107.8MB              2
>>>>>>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>>>>>>        latest              c36d8ff2945d          25 hours ago
>>>>>>>>>  685.6MB             625.1MB               60.52MB             0
>>>>>>>>> <none>
>>>>>>>>>                           <none>              11c86ebe025f        26 hours
>>>>>>>>> ago        1.639GB             922.3MB              716.9MB             0
>>>>>>>>> <none>
>>>>>>>>>                           <none>              2ecd69c89ec1        26 hours
>>>>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test
>>>>>>>>>  latest              3d1d589d44fe        2 days ago          1.305GB
>>>>>>>>>       965.7MB               339.5MB             0
>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test
>>>>>>>>>  latest              d1cc503ebe8e        2 days ago          1.305GB
>>>>>>>>>       965.7MB             339.2MB             0
>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test
>>>>>>>>>  latest              8582c6ca6e15        3 days ago          1.305GB
>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test
>>>>>>>>>  latest              4591e0948170        3 days ago          1.305GB
>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test
>>>>>>>>>  latest              ab181c49d56e        4 days ago          1.305GB
>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test
>>>>>>>>>  latest              2104ba0a6db7        4 days ago          1.305GB
>>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>>> ...
>>>>>>>>> <1000+ images>
>>>>>>>>>
>>>>>>>>> I removed unused the images and the beam15 is back now.
>>>>>>>>>
>>>>>>>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>>>>>>>> Ankur, I assigned the issue to you. Feel free to reassign it if
>>>>>>>>> needed.
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>> Yifan
>>>>>>>>>
>>>>>>>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Something were eating the disk. Disconnected the worker so jobs
>>>>>>>>>> could be allocated to other nodes. Will look deeper.
>>>>>>>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>>>>>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I'm on it.
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Opened a bug here:
>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>>>>>>>
>>>>>>>>>>>> Can someone investigate what's going on?
>>>>>>>>>>>>
>>>>>>>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Mark Liu <ma...@google.com>.
Thank you for triaging and working out a solution Yifan and Ankur.

Ankur, from what you discovered, we should fix this race condition
otherwise same problem will happen in the future. Is there a jira tracking
this issue?

On Fri, Jun 28, 2019 at 4:56 PM Yifan Zou <yi...@google.com> wrote:

> Sorry for the inconvenience. I disabled the worker. I'll need more time to
> restore it.
>
> On Fri, Jun 28, 2019 at 3:56 PM Daniel Oliveira <da...@google.com>
> wrote:
>
>> Any updates to this issue today? It seems like this (or a similar bug) is
>> still happening across many Pre and Postcommits.
>>
>> On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou <yi...@google.com> wrote:
>>
>>> I did the prune on beam15. The disk was free but all jobs fails with
>>> other weird problems. Looks like docker prune overkills, but I don't have
>>> evidence. Will look further in AM.
>>>
>>> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> See how the hdfs IT already avoids tag collisions.
>>>>
>>>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang <zy...@google.com> wrote:
>>>>
>>>>> for flakiness I guess a tag is needed to separate concurrent build
>>>>> apart.
>>>>>
>>>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com> wrote:
>>>>>
>>>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> This highlights the race condition caused by using single docker
>>>>>>> registry on a machine.
>>>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>>>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>>>>>>
>>>>>>> Is their a way to dynamically create and destroy docker repository
>>>>>>> on a machine and clean all the relevant data?
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The problem was because of the large quantity of stale docker
>>>>>>>> images generated by the Python portable tests and HDFS IT.
>>>>>>>>
>>>>>>>> Dumping the docker disk usage gives me:
>>>>>>>>
>>>>>>>> TYPE                TOTAL               ACTIVE              SIZE
>>>>>>>>              RECLAIMABLE
>>>>>>>> *Images              1039                356                 424GB
>>>>>>>>               384.2GB (90%)*
>>>>>>>> Containers          987                 2                   2.042GB
>>>>>>>>             2.041GB (99%)
>>>>>>>> Local Volumes       126                 0                   392.8MB
>>>>>>>>             392.8MB (100%)
>>>>>>>>
>>>>>>>> REPOSITORY
>>>>>>>>                   TAG                 IMAGE ID            CREATED
>>>>>>>>   SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>>>    latest              ff1b949f4442        22 hours ago        1.639GB
>>>>>>>>         922.3MB                  716.9MB             0
>>>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>>>      latest              1dda7b9d9748        22 hours ago        1.624GB
>>>>>>>>           913.7MB               710.3MB             0
>>>>>>>> <none>
>>>>>>>>                          <none>              05458187a0e3        22 hours
>>>>>>>> ago        732.9MB             625.1MB            107.8MB             4
>>>>>>>> <none>
>>>>>>>>                          <none>              896f35dd685f        23 hours
>>>>>>>> ago        1.639GB             922.3MB               716.9MB             0
>>>>>>>> <none>
>>>>>>>>                          <none>              db4d24ca9f2b        23 hours
>>>>>>>> ago        1.624GB             913.7MB              710.3MB             0
>>>>>>>> <none>
>>>>>>>>                           <none>              547df4d71c31        23 hours
>>>>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>>>>> <none>
>>>>>>>>                           <none>              dd7d9582c3e0        23 hours
>>>>>>>> ago        1.639GB             922.3MB             716.9MB             0
>>>>>>>> <none>
>>>>>>>>                           <none>              664aae255239        23 hours
>>>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>>>> <none>
>>>>>>>>                           <none>              b528fedf9228        23 hours
>>>>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>>>>> <none>
>>>>>>>>                           <none>              8e996f22435e        25 hours
>>>>>>>> ago        1.624GB             913.7MB            710.3MB             0
>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>>>>>>>>              24b73b3fec06        25 hours ago        1.305GB
>>>>>>>> 965.7MB               339.5MB             0
>>>>>>>> <none>
>>>>>>>>                           <none>              096325fb48de       25 hours
>>>>>>>> ago        732.9MB             625.1MB            107.8MB              2
>>>>>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>>>>>        latest              c36d8ff2945d          25 hours ago
>>>>>>>>  685.6MB             625.1MB               60.52MB             0
>>>>>>>> <none>
>>>>>>>>                           <none>              11c86ebe025f        26 hours
>>>>>>>> ago        1.639GB             922.3MB              716.9MB             0
>>>>>>>> <none>
>>>>>>>>                           <none>              2ecd69c89ec1        26 hours
>>>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test
>>>>>>>>  latest              3d1d589d44fe        2 days ago          1.305GB
>>>>>>>>       965.7MB               339.5MB             0
>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test
>>>>>>>>  latest              d1cc503ebe8e        2 days ago          1.305GB
>>>>>>>>       965.7MB             339.2MB             0
>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test
>>>>>>>>  latest              8582c6ca6e15        3 days ago          1.305GB
>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test
>>>>>>>>  latest              4591e0948170        3 days ago          1.305GB
>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test
>>>>>>>>  latest              ab181c49d56e        4 days ago          1.305GB
>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test
>>>>>>>>  latest              2104ba0a6db7        4 days ago          1.305GB
>>>>>>>>       965.7MB              339.2MB             0
>>>>>>>> ...
>>>>>>>> <1000+ images>
>>>>>>>>
>>>>>>>> I removed unused the images and the beam15 is back now.
>>>>>>>>
>>>>>>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>>>>>>> Ankur, I assigned the issue to you. Feel free to reassign it if
>>>>>>>> needed.
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>> Yifan
>>>>>>>>
>>>>>>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Something were eating the disk. Disconnected the worker so jobs
>>>>>>>>> could be allocated to other nodes. Will look deeper.
>>>>>>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>>>>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I'm on it.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Opened a bug here:
>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>>>>>>
>>>>>>>>>>> Can someone investigate what's going on?
>>>>>>>>>>>
>>>>>>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Yifan Zou <yi...@google.com>.
Sorry for the inconvenience. I disabled the worker. I'll need more time to
restore it.

On Fri, Jun 28, 2019 at 3:56 PM Daniel Oliveira <da...@google.com>
wrote:

> Any updates to this issue today? It seems like this (or a similar bug) is
> still happening across many Pre and Postcommits.
>
> On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou <yi...@google.com> wrote:
>
>> I did the prune on beam15. The disk was free but all jobs fails with
>> other weird problems. Looks like docker prune overkills, but I don't have
>> evidence. Will look further in AM.
>>
>> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri <eh...@google.com> wrote:
>>
>>> See how the hdfs IT already avoids tag collisions.
>>>
>>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang <zy...@google.com> wrote:
>>>
>>>> for flakiness I guess a tag is needed to separate concurrent build
>>>> apart.
>>>>
>>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com> wrote:
>>>>
>>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>>
>>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com>
>>>>> wrote:
>>>>>
>>>>>> This highlights the race condition caused by using single docker
>>>>>> registry on a machine.
>>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>>>>>
>>>>>> Is their a way to dynamically create and destroy docker repository on
>>>>>> a machine and clean all the relevant data?
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> The problem was because of the large quantity of stale docker images
>>>>>>> generated by the Python portable tests and HDFS IT.
>>>>>>>
>>>>>>> Dumping the docker disk usage gives me:
>>>>>>>
>>>>>>> TYPE                TOTAL               ACTIVE              SIZE
>>>>>>>            RECLAIMABLE
>>>>>>> *Images              1039                356                 424GB
>>>>>>>             384.2GB (90%)*
>>>>>>> Containers          987                 2                   2.042GB
>>>>>>>             2.041GB (99%)
>>>>>>> Local Volumes       126                 0                   392.8MB
>>>>>>>             392.8MB (100%)
>>>>>>>
>>>>>>> REPOSITORY
>>>>>>>                   TAG                 IMAGE ID            CREATED
>>>>>>>   SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>>    latest              ff1b949f4442        22 hours ago        1.639GB
>>>>>>>         922.3MB                  716.9MB             0
>>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>>    latest              1dda7b9d9748        22 hours ago        1.624GB
>>>>>>>         913.7MB               710.3MB             0
>>>>>>> <none>
>>>>>>>                          <none>              05458187a0e3        22 hours
>>>>>>> ago        732.9MB             625.1MB            107.8MB             4
>>>>>>> <none>
>>>>>>>                          <none>              896f35dd685f        23 hours
>>>>>>> ago        1.639GB             922.3MB               716.9MB             0
>>>>>>> <none>
>>>>>>>                          <none>              db4d24ca9f2b        23 hours
>>>>>>> ago        1.624GB             913.7MB              710.3MB             0
>>>>>>> <none>
>>>>>>>                         <none>              547df4d71c31        23 hours
>>>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>>>> <none>
>>>>>>>                         <none>              dd7d9582c3e0        23 hours
>>>>>>> ago        1.639GB             922.3MB             716.9MB             0
>>>>>>> <none>
>>>>>>>                         <none>              664aae255239        23 hours
>>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>>> <none>
>>>>>>>                           <none>              b528fedf9228        23 hours
>>>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>>>> <none>
>>>>>>>                           <none>              8e996f22435e        25 hours
>>>>>>> ago        1.624GB             913.7MB            710.3MB             0
>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>>>>>>>              24b73b3fec06        25 hours ago        1.305GB
>>>>>>> 965.7MB               339.5MB             0
>>>>>>> <none>
>>>>>>>                           <none>              096325fb48de       25 hours
>>>>>>> ago        732.9MB             625.1MB            107.8MB              2
>>>>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>>>>      latest              c36d8ff2945d          25 hours ago        685.6MB
>>>>>>>             625.1MB               60.52MB             0
>>>>>>> <none>
>>>>>>>                         <none>              11c86ebe025f        26 hours
>>>>>>> ago        1.639GB             922.3MB              716.9MB             0
>>>>>>> <none>
>>>>>>>                         <none>              2ecd69c89ec1        26 hours
>>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test
>>>>>>>  latest              3d1d589d44fe        2 days ago          1.305GB
>>>>>>>       965.7MB               339.5MB             0
>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test
>>>>>>>  latest              d1cc503ebe8e        2 days ago          1.305GB
>>>>>>>       965.7MB             339.2MB             0
>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test
>>>>>>>  latest              8582c6ca6e15        3 days ago          1.305GB
>>>>>>>       965.7MB              339.2MB             0
>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test
>>>>>>>  latest              4591e0948170        3 days ago          1.305GB
>>>>>>>       965.7MB              339.2MB             0
>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test
>>>>>>>  latest              ab181c49d56e        4 days ago          1.305GB
>>>>>>>       965.7MB              339.2MB             0
>>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test
>>>>>>>  latest              2104ba0a6db7        4 days ago          1.305GB
>>>>>>>       965.7MB              339.2MB             0
>>>>>>> ...
>>>>>>> <1000+ images>
>>>>>>>
>>>>>>> I removed unused the images and the beam15 is back now.
>>>>>>>
>>>>>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>>>>>> Ankur, I assigned the issue to you. Feel free to reassign it if
>>>>>>> needed.
>>>>>>>
>>>>>>> Thank you.
>>>>>>> Yifan
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Something were eating the disk. Disconnected the worker so jobs
>>>>>>>> could be allocated to other nodes. Will look deeper.
>>>>>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>>>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'm on it.
>>>>>>>>>
>>>>>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Opened a bug here:
>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>>>>>
>>>>>>>>>> Can someone investigate what's going on?
>>>>>>>>>>
>>>>>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Daniel Oliveira <da...@google.com>.
Any updates to this issue today? It seems like this (or a similar bug) is
still happening across many Pre and Postcommits.

On Fri, Jun 28, 2019 at 12:33 AM Yifan Zou <yi...@google.com> wrote:

> I did the prune on beam15. The disk was free but all jobs fails with other
> weird problems. Looks like docker prune overkills, but I don't have
> evidence. Will look further in AM.
>
> On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri <eh...@google.com> wrote:
>
>> See how the hdfs IT already avoids tag collisions.
>>
>> On Thu, Jun 27, 2019, 20:42 Yichi Zhang <zy...@google.com> wrote:
>>
>>> for flakiness I guess a tag is needed to separate concurrent build
>>> apart.
>>>
>>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com> wrote:
>>>
>>>> maybe a cron job on jenkins node that does docker prune every day?
>>>>
>>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com> wrote:
>>>>
>>>>> This highlights the race condition caused by using single docker
>>>>> registry on a machine.
>>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>>>>
>>>>> Is their a way to dynamically create and destroy docker repository on
>>>>> a machine and clean all the relevant data?
>>>>>
>>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com> wrote:
>>>>>
>>>>>> The problem was because of the large quantity of stale docker images
>>>>>> generated by the Python portable tests and HDFS IT.
>>>>>>
>>>>>> Dumping the docker disk usage gives me:
>>>>>>
>>>>>> TYPE                TOTAL               ACTIVE              SIZE
>>>>>>            RECLAIMABLE
>>>>>> *Images              1039                356                 424GB
>>>>>>             384.2GB (90%)*
>>>>>> Containers          987                 2                   2.042GB
>>>>>>           2.041GB (99%)
>>>>>> Local Volumes       126                 0                   392.8MB
>>>>>>           392.8MB (100%)
>>>>>>
>>>>>> REPOSITORY
>>>>>>                 TAG                 IMAGE ID            CREATED
>>>>>> SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>>  latest              ff1b949f4442        22 hours ago        1.639GB
>>>>>>       922.3MB                  716.9MB             0
>>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>>    latest              1dda7b9d9748        22 hours ago        1.624GB
>>>>>>         913.7MB               710.3MB             0
>>>>>> <none>
>>>>>>                        <none>              05458187a0e3        22 hours ago
>>>>>>        732.9MB             625.1MB            107.8MB             4
>>>>>> <none>
>>>>>>                        <none>              896f35dd685f        23 hours ago
>>>>>>        1.639GB             922.3MB               716.9MB             0
>>>>>> <none>
>>>>>>                        <none>              db4d24ca9f2b        23 hours ago
>>>>>>        1.624GB             913.7MB              710.3MB             0
>>>>>> <none>
>>>>>>                         <none>              547df4d71c31        23 hours
>>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>>> <none>
>>>>>>                         <none>              dd7d9582c3e0        23 hours
>>>>>> ago        1.639GB             922.3MB             716.9MB             0
>>>>>> <none>
>>>>>>                         <none>              664aae255239        23 hours
>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>> <none>
>>>>>>                         <none>              b528fedf9228        23 hours
>>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>>> <none>
>>>>>>                         <none>              8e996f22435e        25 hours
>>>>>> ago        1.624GB             913.7MB            710.3MB             0
>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>>>>>>            24b73b3fec06        25 hours ago        1.305GB
>>>>>> 965.7MB               339.5MB             0
>>>>>> <none>
>>>>>>                         <none>              096325fb48de       25 hours ago
>>>>>>        732.9MB             625.1MB            107.8MB              2
>>>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>>>      latest              c36d8ff2945d          25 hours ago        685.6MB
>>>>>>             625.1MB               60.52MB             0
>>>>>> <none>
>>>>>>                         <none>              11c86ebe025f        26 hours
>>>>>> ago        1.639GB             922.3MB              716.9MB             0
>>>>>> <none>
>>>>>>                         <none>              2ecd69c89ec1        26 hours
>>>>>> ago        1.624GB             913.7MB             710.3MB             0
>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test        latest
>>>>>>              3d1d589d44fe        2 days ago          1.305GB
>>>>>> 965.7MB               339.5MB             0
>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test      latest
>>>>>>              d1cc503ebe8e        2 days ago          1.305GB
>>>>>> 965.7MB             339.2MB             0
>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test        latest
>>>>>>              8582c6ca6e15        3 days ago          1.305GB
>>>>>> 965.7MB              339.2MB             0
>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test        latest
>>>>>>              4591e0948170        3 days ago          1.305GB
>>>>>> 965.7MB              339.2MB             0
>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test        latest
>>>>>>              ab181c49d56e        4 days ago          1.305GB
>>>>>> 965.7MB              339.2MB             0
>>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test        latest
>>>>>>              2104ba0a6db7        4 days ago          1.305GB
>>>>>> 965.7MB              339.2MB             0
>>>>>> ...
>>>>>> <1000+ images>
>>>>>>
>>>>>> I removed unused the images and the beam15 is back now.
>>>>>>
>>>>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>>>>> Ankur, I assigned the issue to you. Feel free to reassign it if
>>>>>> needed.
>>>>>>
>>>>>> Thank you.
>>>>>> Yifan
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Something were eating the disk. Disconnected the worker so jobs
>>>>>>> could be allocated to other nodes. Will look deeper.
>>>>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I'm on it.
>>>>>>>>
>>>>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>>>>
>>>>>>>>> Can someone investigate what's going on?
>>>>>>>>>
>>>>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Yifan Zou <yi...@google.com>.
I did the prune on beam15. The disk was free but all jobs fails with other
weird problems. Looks like docker prune overkills, but I don't have
evidence. Will look further in AM.

On Thu, Jun 27, 2019 at 11:20 PM Udi Meiri <eh...@google.com> wrote:

> See how the hdfs IT already avoids tag collisions.
>
> On Thu, Jun 27, 2019, 20:42 Yichi Zhang <zy...@google.com> wrote:
>
>> for flakiness I guess a tag is needed to separate concurrent build apart.
>>
>> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com> wrote:
>>
>>> maybe a cron job on jenkins node that does docker prune every day?
>>>
>>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com> wrote:
>>>
>>>> This highlights the race condition caused by using single docker
>>>> registry on a machine.
>>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>>>
>>>> Is their a way to dynamically create and destroy docker repository on a
>>>> machine and clean all the relevant data?
>>>>
>>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com> wrote:
>>>>
>>>>> The problem was because of the large quantity of stale docker images
>>>>> generated by the Python portable tests and HDFS IT.
>>>>>
>>>>> Dumping the docker disk usage gives me:
>>>>>
>>>>> TYPE                TOTAL               ACTIVE              SIZE
>>>>>          RECLAIMABLE
>>>>> *Images              1039                356                 424GB
>>>>>           384.2GB (90%)*
>>>>> Containers          987                 2                   2.042GB
>>>>>           2.041GB (99%)
>>>>> Local Volumes       126                 0                   392.8MB
>>>>>           392.8MB (100%)
>>>>>
>>>>> REPOSITORY
>>>>>                 TAG                 IMAGE ID            CREATED
>>>>> SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>>  latest              ff1b949f4442        22 hours ago        1.639GB
>>>>>       922.3MB                  716.9MB             0
>>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>>  latest              1dda7b9d9748        22 hours ago        1.624GB
>>>>>       913.7MB               710.3MB             0
>>>>> <none>
>>>>>                        <none>              05458187a0e3        22 hours ago
>>>>>        732.9MB             625.1MB            107.8MB             4
>>>>> <none>
>>>>>                        <none>              896f35dd685f        23 hours ago
>>>>>        1.639GB             922.3MB               716.9MB             0
>>>>> <none>
>>>>>                        <none>              db4d24ca9f2b        23 hours ago
>>>>>        1.624GB             913.7MB              710.3MB             0
>>>>> <none>
>>>>>                       <none>              547df4d71c31        23 hours ago
>>>>>        732.9MB             625.1MB             107.8MB             4
>>>>> <none>
>>>>>                       <none>              dd7d9582c3e0        23 hours ago
>>>>>        1.639GB             922.3MB             716.9MB             0
>>>>> <none>
>>>>>                       <none>              664aae255239        23 hours ago
>>>>>        1.624GB             913.7MB             710.3MB             0
>>>>> <none>
>>>>>                         <none>              b528fedf9228        23 hours
>>>>> ago        732.9MB             625.1MB             107.8MB             4
>>>>> <none>
>>>>>                         <none>              8e996f22435e        25 hours
>>>>> ago        1.624GB             913.7MB            710.3MB             0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>>>>>            24b73b3fec06        25 hours ago        1.305GB
>>>>> 965.7MB               339.5MB             0
>>>>> <none>
>>>>>                         <none>              096325fb48de       25 hours ago
>>>>>        732.9MB             625.1MB            107.8MB              2
>>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>>    latest              c36d8ff2945d          25 hours ago        685.6MB
>>>>>           625.1MB               60.52MB             0
>>>>> <none>
>>>>>                       <none>              11c86ebe025f        26 hours ago
>>>>>        1.639GB             922.3MB              716.9MB             0
>>>>> <none>
>>>>>                       <none>              2ecd69c89ec1        26 hours ago
>>>>>        1.624GB             913.7MB             710.3MB             0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test        latest
>>>>>              3d1d589d44fe        2 days ago          1.305GB
>>>>> 965.7MB               339.5MB             0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test      latest
>>>>>              d1cc503ebe8e        2 days ago          1.305GB
>>>>> 965.7MB             339.2MB             0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test        latest
>>>>>              8582c6ca6e15        3 days ago          1.305GB
>>>>> 965.7MB              339.2MB             0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test        latest
>>>>>              4591e0948170        3 days ago          1.305GB
>>>>> 965.7MB              339.2MB             0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test        latest
>>>>>              ab181c49d56e        4 days ago          1.305GB
>>>>> 965.7MB              339.2MB             0
>>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test        latest
>>>>>              2104ba0a6db7        4 days ago          1.305GB
>>>>> 965.7MB              339.2MB             0
>>>>> ...
>>>>> <1000+ images>
>>>>>
>>>>> I removed unused the images and the beam15 is back now.
>>>>>
>>>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>>>> Ankur, I assigned the issue to you. Feel free to reassign it if
>>>>> needed.
>>>>>
>>>>> Thank you.
>>>>> Yifan
>>>>>
>>>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Something were eating the disk. Disconnected the worker so jobs could
>>>>>> be allocated to other nodes. Will look deeper.
>>>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm on it.
>>>>>>>
>>>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>>
>>>>>>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>>>
>>>>>>>> Can someone investigate what's going on?
>>>>>>>>
>>>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Udi Meiri <eh...@google.com>.
See how the hdfs IT already avoids tag collisions.

On Thu, Jun 27, 2019, 20:42 Yichi Zhang <zy...@google.com> wrote:

> for flakiness I guess a tag is needed to separate concurrent build apart.
>
> On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com> wrote:
>
>> maybe a cron job on jenkins node that does docker prune every day?
>>
>> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com> wrote:
>>
>>> This highlights the race condition caused by using single docker
>>> registry on a machine.
>>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>>
>>> Is their a way to dynamically create and destroy docker repository on a
>>> machine and clean all the relevant data?
>>>
>>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com> wrote:
>>>
>>>> The problem was because of the large quantity of stale docker images
>>>> generated by the Python portable tests and HDFS IT.
>>>>
>>>> Dumping the docker disk usage gives me:
>>>>
>>>> TYPE                TOTAL               ACTIVE              SIZE
>>>>          RECLAIMABLE
>>>> *Images              1039                356                 424GB
>>>>           384.2GB (90%)*
>>>> Containers          987                 2                   2.042GB
>>>>         2.041GB (99%)
>>>> Local Volumes       126                 0                   392.8MB
>>>>         392.8MB (100%)
>>>>
>>>> REPOSITORY
>>>>               TAG                 IMAGE ID            CREATED
>>>> SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>>> jenkins-docker-apache.bintray.io/beam/python3
>>>>  latest              ff1b949f4442        22 hours ago        1.639GB
>>>>       922.3MB                  716.9MB             0
>>>> jenkins-docker-apache.bintray.io/beam/python
>>>>  latest              1dda7b9d9748        22 hours ago        1.624GB
>>>>       913.7MB               710.3MB             0
>>>> <none>
>>>>                      <none>              05458187a0e3        22 hours ago
>>>>      732.9MB             625.1MB            107.8MB             4
>>>> <none>
>>>>                      <none>              896f35dd685f        23 hours ago
>>>>      1.639GB             922.3MB               716.9MB             0
>>>> <none>
>>>>                      <none>              db4d24ca9f2b        23 hours ago
>>>>      1.624GB             913.7MB              710.3MB             0
>>>> <none>
>>>>                       <none>              547df4d71c31        23 hours ago
>>>>        732.9MB             625.1MB             107.8MB             4
>>>> <none>
>>>>                       <none>              dd7d9582c3e0        23 hours ago
>>>>        1.639GB             922.3MB             716.9MB             0
>>>> <none>
>>>>                       <none>              664aae255239        23 hours ago
>>>>        1.624GB             913.7MB             710.3MB             0
>>>> <none>
>>>>                       <none>              b528fedf9228        23 hours ago
>>>>        732.9MB             625.1MB             107.8MB             4
>>>> <none>
>>>>                       <none>              8e996f22435e        25 hours ago
>>>>        1.624GB             913.7MB            710.3MB             0
>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>>>>          24b73b3fec06        25 hours ago        1.305GB
>>>> 965.7MB               339.5MB             0
>>>> <none>
>>>>                       <none>              096325fb48de       25 hours ago
>>>>      732.9MB             625.1MB            107.8MB              2
>>>> jenkins-docker-apache.bintray.io/beam/java
>>>>    latest              c36d8ff2945d          25 hours ago        685.6MB
>>>>           625.1MB               60.52MB             0
>>>> <none>
>>>>                       <none>              11c86ebe025f        26 hours ago
>>>>        1.639GB             922.3MB              716.9MB             0
>>>> <none>
>>>>                       <none>              2ecd69c89ec1        26 hours ago
>>>>        1.624GB             913.7MB             710.3MB             0
>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test        latest
>>>>            3d1d589d44fe        2 days ago          1.305GB
>>>> 965.7MB               339.5MB             0
>>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test      latest
>>>>            d1cc503ebe8e        2 days ago          1.305GB
>>>> 965.7MB             339.2MB             0
>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test        latest
>>>>            8582c6ca6e15        3 days ago          1.305GB
>>>> 965.7MB              339.2MB             0
>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test        latest
>>>>            4591e0948170        3 days ago          1.305GB
>>>> 965.7MB              339.2MB             0
>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test        latest
>>>>            ab181c49d56e        4 days ago          1.305GB
>>>> 965.7MB              339.2MB             0
>>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test        latest
>>>>            2104ba0a6db7        4 days ago          1.305GB
>>>> 965.7MB              339.2MB             0
>>>> ...
>>>> <1000+ images>
>>>>
>>>> I removed unused the images and the beam15 is back now.
>>>>
>>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>>> Ankur, I assigned the issue to you. Feel free to reassign it if needed.
>>>>
>>>> Thank you.
>>>> Yifan
>>>>
>>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com> wrote:
>>>>
>>>>> Something were eating the disk. Disconnected the worker so jobs could
>>>>> be allocated to other nodes. Will look deeper.
>>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>>
>>>>>
>>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I'm on it.
>>>>>>
>>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:
>>>>>>
>>>>>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>>
>>>>>>> Can someone investigate what's going on?
>>>>>>>
>>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Yichi Zhang <zy...@google.com>.
for flakiness I guess a tag is needed to separate concurrent build apart.

On Thu, Jun 27, 2019 at 8:39 PM Yichi Zhang <zy...@google.com> wrote:

> maybe a cron job on jenkins node that does docker prune every day?
>
> On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com> wrote:
>
>> This highlights the race condition caused by using single docker registry
>> on a machine.
>> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
>> after another then the 2nd one will replace the 1st one and cause flakyness.
>>
>> Is their a way to dynamically create and destroy docker repository on a
>> machine and clean all the relevant data?
>>
>> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com> wrote:
>>
>>> The problem was because of the large quantity of stale docker images
>>> generated by the Python portable tests and HDFS IT.
>>>
>>> Dumping the docker disk usage gives me:
>>>
>>> TYPE                TOTAL               ACTIVE              SIZE
>>>        RECLAIMABLE
>>> *Images              1039                356                 424GB
>>>         384.2GB (90%)*
>>> Containers          987                 2                   2.042GB
>>>         2.041GB (99%)
>>> Local Volumes       126                 0                   392.8MB
>>>         392.8MB (100%)
>>>
>>> REPOSITORY
>>>               TAG                 IMAGE ID            CREATED
>>> SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>>> jenkins-docker-apache.bintray.io/beam/python3
>>>  latest              ff1b949f4442        22 hours ago        1.639GB
>>>       922.3MB                  716.9MB             0
>>> jenkins-docker-apache.bintray.io/beam/python
>>>  latest              1dda7b9d9748        22 hours ago        1.624GB
>>>       913.7MB               710.3MB             0
>>> <none>
>>>                      <none>              05458187a0e3        22 hours ago
>>>      732.9MB             625.1MB            107.8MB             4
>>> <none>
>>>                      <none>              896f35dd685f        23 hours ago
>>>      1.639GB             922.3MB               716.9MB             0
>>> <none>
>>>                      <none>              db4d24ca9f2b        23 hours ago
>>>      1.624GB             913.7MB              710.3MB             0
>>> <none>
>>>                     <none>              547df4d71c31        23 hours ago
>>>      732.9MB             625.1MB             107.8MB             4
>>> <none>
>>>                     <none>              dd7d9582c3e0        23 hours ago
>>>      1.639GB             922.3MB             716.9MB             0
>>> <none>
>>>                     <none>              664aae255239        23 hours ago
>>>      1.624GB             913.7MB             710.3MB             0
>>> <none>
>>>                       <none>              b528fedf9228        23 hours ago
>>>        732.9MB             625.1MB             107.8MB             4
>>> <none>
>>>                       <none>              8e996f22435e        25 hours ago
>>>        1.624GB             913.7MB            710.3MB             0
>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>>>          24b73b3fec06        25 hours ago        1.305GB
>>> 965.7MB               339.5MB             0
>>> <none>
>>>                       <none>              096325fb48de       25 hours ago
>>>      732.9MB             625.1MB            107.8MB              2
>>> jenkins-docker-apache.bintray.io/beam/java
>>>  latest              c36d8ff2945d          25 hours ago        685.6MB
>>>         625.1MB               60.52MB             0
>>> <none>
>>>                     <none>              11c86ebe025f        26 hours ago
>>>      1.639GB             922.3MB              716.9MB             0
>>> <none>
>>>                     <none>              2ecd69c89ec1        26 hours ago
>>>      1.624GB             913.7MB             710.3MB             0
>>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test        latest
>>>            3d1d589d44fe        2 days ago          1.305GB
>>> 965.7MB               339.5MB             0
>>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test      latest
>>>            d1cc503ebe8e        2 days ago          1.305GB
>>> 965.7MB             339.2MB             0
>>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test        latest
>>>            8582c6ca6e15        3 days ago          1.305GB
>>> 965.7MB              339.2MB             0
>>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test        latest
>>>            4591e0948170        3 days ago          1.305GB
>>> 965.7MB              339.2MB             0
>>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test        latest
>>>            ab181c49d56e        4 days ago          1.305GB
>>> 965.7MB              339.2MB             0
>>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test        latest
>>>            2104ba0a6db7        4 days ago          1.305GB
>>> 965.7MB              339.2MB             0
>>> ...
>>> <1000+ images>
>>>
>>> I removed unused the images and the beam15 is back now.
>>>
>>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>>> Ankur, I assigned the issue to you. Feel free to reassign it if needed.
>>>
>>> Thank you.
>>> Yifan
>>>
>>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com> wrote:
>>>
>>>> Something were eating the disk. Disconnected the worker so jobs could
>>>> be allocated to other nodes. Will look deeper.
>>>> Filesystem      Size  Used  Avail Use% Mounted on
>>>> /dev/sda1       485G  485G 96K 100%  /
>>>>
>>>>
>>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com> wrote:
>>>>
>>>>> I'm on it.
>>>>>
>>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:
>>>>>
>>>>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>>>>
>>>>>> Can someone investigate what's going on?
>>>>>>
>>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Yichi Zhang <zy...@google.com>.
maybe a cron job on jenkins node that does docker prune every day?

On Thu, Jun 27, 2019 at 6:58 PM Ankur Goenka <go...@google.com> wrote:

> This highlights the race condition caused by using single docker registry
> on a machine.
> If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one
> after another then the 2nd one will replace the 1st one and cause flakyness.
>
> Is their a way to dynamically create and destroy docker repository on a
> machine and clean all the relevant data?
>
> On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com> wrote:
>
>> The problem was because of the large quantity of stale docker images
>> generated by the Python portable tests and HDFS IT.
>>
>> Dumping the docker disk usage gives me:
>>
>> TYPE                TOTAL               ACTIVE              SIZE
>>        RECLAIMABLE
>> *Images              1039                356                 424GB
>>         384.2GB (90%)*
>> Containers          987                 2                   2.042GB
>>       2.041GB (99%)
>> Local Volumes       126                 0                   392.8MB
>>       392.8MB (100%)
>>
>> REPOSITORY
>>             TAG                 IMAGE ID            CREATED
>> SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
>> jenkins-docker-apache.bintray.io/beam/python3
>>  latest              ff1b949f4442        22 hours ago        1.639GB
>>       922.3MB                  716.9MB             0
>> jenkins-docker-apache.bintray.io/beam/python
>>  latest              1dda7b9d9748        22 hours ago        1.624GB
>>       913.7MB               710.3MB             0
>> <none>
>>                    <none>              05458187a0e3        22 hours ago
>>    732.9MB             625.1MB            107.8MB             4
>> <none>
>>                    <none>              896f35dd685f        23 hours ago
>>    1.639GB             922.3MB               716.9MB             0
>> <none>
>>                    <none>              db4d24ca9f2b        23 hours ago
>>    1.624GB             913.7MB              710.3MB             0
>> <none>
>>                     <none>              547df4d71c31        23 hours ago
>>      732.9MB             625.1MB             107.8MB             4
>> <none>
>>                     <none>              dd7d9582c3e0        23 hours ago
>>      1.639GB             922.3MB             716.9MB             0
>> <none>
>>                     <none>              664aae255239        23 hours ago
>>      1.624GB             913.7MB             710.3MB             0
>> <none>
>>                     <none>              b528fedf9228        23 hours ago
>>      732.9MB             625.1MB             107.8MB             4
>> <none>
>>                     <none>              8e996f22435e        25 hours ago
>>      1.624GB             913.7MB            710.3MB             0
>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>>        24b73b3fec06        25 hours ago        1.305GB             965.7MB
>>              339.5MB             0
>> <none>
>>                     <none>              096325fb48de       25 hours ago
>>    732.9MB             625.1MB            107.8MB              2
>> jenkins-docker-apache.bintray.io/beam/java
>>  latest              c36d8ff2945d          25 hours ago        685.6MB
>>         625.1MB               60.52MB             0
>> <none>
>>                     <none>              11c86ebe025f        26 hours ago
>>      1.639GB             922.3MB              716.9MB             0
>> <none>
>>                     <none>              2ecd69c89ec1        26 hours ago
>>      1.624GB             913.7MB             710.3MB             0
>> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test        latest
>>          3d1d589d44fe        2 days ago          1.305GB
>> 965.7MB               339.5MB             0
>> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test      latest
>>          d1cc503ebe8e        2 days ago          1.305GB
>> 965.7MB             339.2MB             0
>> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test        latest
>>          8582c6ca6e15        3 days ago          1.305GB
>> 965.7MB              339.2MB             0
>> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test        latest
>>          4591e0948170        3 days ago          1.305GB
>> 965.7MB              339.2MB             0
>> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test        latest
>>          ab181c49d56e        4 days ago          1.305GB
>> 965.7MB              339.2MB             0
>> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test        latest
>>          2104ba0a6db7        4 days ago          1.305GB
>> 965.7MB              339.2MB             0
>> ...
>> <1000+ images>
>>
>> I removed unused the images and the beam15 is back now.
>>
>> Opened https://issues.apache.org/jira/browse/BEAM-7650.
>> Ankur, I assigned the issue to you. Feel free to reassign it if needed.
>>
>> Thank you.
>> Yifan
>>
>> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com> wrote:
>>
>>> Something were eating the disk. Disconnected the worker so jobs could be
>>> allocated to other nodes. Will look deeper.
>>> Filesystem      Size  Used  Avail Use% Mounted on
>>> /dev/sda1       485G  485G 96K 100%  /
>>>
>>>
>>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com> wrote:
>>>
>>>> I'm on it.
>>>>
>>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:
>>>>
>>>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>>>
>>>>> Can someone investigate what's going on?
>>>>>
>>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Ankur Goenka <go...@google.com>.
This highlights the race condition caused by using single docker registry
on a machine.
If 2 tests create "jenkins-docker-apache.bintray.io/beam/python" one after
another then the 2nd one will replace the 1st one and cause flakyness.

Is their a way to dynamically create and destroy docker repository on a
machine and clean all the relevant data?

On Thu, Jun 27, 2019 at 3:15 PM Yifan Zou <yi...@google.com> wrote:

> The problem was because of the large quantity of stale docker images
> generated by the Python portable tests and HDFS IT.
>
> Dumping the docker disk usage gives me:
>
> TYPE                TOTAL               ACTIVE              SIZE
>      RECLAIMABLE
> *Images              1039                356                 424GB
>       384.2GB (90%)*
> Containers          987                 2                   2.042GB
>       2.041GB (99%)
> Local Volumes       126                 0                   392.8MB
>       392.8MB (100%)
>
> REPOSITORY
>             TAG                 IMAGE ID            CREATED
> SIZE                SHARED SIZE         UNIQUE SIZE         CONTAINERS
> jenkins-docker-apache.bintray.io/beam/python3
>  latest              ff1b949f4442        22 hours ago        1.639GB
>       922.3MB                  716.9MB             0
> jenkins-docker-apache.bintray.io/beam/python
>  latest              1dda7b9d9748        22 hours ago        1.624GB
>       913.7MB               710.3MB             0
> <none>
>                    <none>              05458187a0e3        22 hours ago
>    732.9MB             625.1MB            107.8MB             4
> <none>
>                    <none>              896f35dd685f        23 hours ago
>    1.639GB             922.3MB               716.9MB             0
> <none>
>                    <none>              db4d24ca9f2b        23 hours ago
>    1.624GB             913.7MB              710.3MB             0
> <none>
>                   <none>              547df4d71c31        23 hours ago
>    732.9MB             625.1MB             107.8MB             4
> <none>
>                   <none>              dd7d9582c3e0        23 hours ago
>    1.639GB             922.3MB             716.9MB             0
> <none>
>                   <none>              664aae255239        23 hours ago
>    1.624GB             913.7MB             710.3MB             0
> <none>
>                     <none>              b528fedf9228        23 hours ago
>      732.9MB             625.1MB             107.8MB             4
> <none>
>                     <none>              8e996f22435e        25 hours ago
>      1.624GB             913.7MB            710.3MB             0
> hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
>        24b73b3fec06        25 hours ago        1.305GB             965.7MB
>              339.5MB             0
> <none>
>                     <none>              096325fb48de       25 hours ago
>    732.9MB             625.1MB            107.8MB              2
> jenkins-docker-apache.bintray.io/beam/java
>  latest              c36d8ff2945d          25 hours ago        685.6MB
>         625.1MB               60.52MB             0
> <none>
>                   <none>              11c86ebe025f        26 hours ago
>    1.639GB             922.3MB              716.9MB             0
> <none>
>                   <none>              2ecd69c89ec1        26 hours ago
>    1.624GB             913.7MB             710.3MB             0
> hdfs_it-jenkins-beam_postcommit_python_verify-8590_test        latest
>          3d1d589d44fe        2 days ago          1.305GB
> 965.7MB               339.5MB             0
> hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test      latest
>          d1cc503ebe8e        2 days ago          1.305GB
> 965.7MB             339.2MB             0
> hdfs_it-jenkins-beam_postcommit_python_verify-8577_test        latest
>          8582c6ca6e15        3 days ago          1.305GB
> 965.7MB              339.2MB             0
> hdfs_it-jenkins-beam_postcommit_python_verify-8576_test        latest
>          4591e0948170        3 days ago          1.305GB
> 965.7MB              339.2MB             0
> hdfs_it-jenkins-beam_postcommit_python_verify-8575_test        latest
>          ab181c49d56e        4 days ago          1.305GB
> 965.7MB              339.2MB             0
> hdfs_it-jenkins-beam_postcommit_python_verify-8573_test        latest
>          2104ba0a6db7        4 days ago          1.305GB
> 965.7MB              339.2MB             0
> ...
> <1000+ images>
>
> I removed unused the images and the beam15 is back now.
>
> Opened https://issues.apache.org/jira/browse/BEAM-7650.
> Ankur, I assigned the issue to you. Feel free to reassign it if needed.
>
> Thank you.
> Yifan
>
> On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com> wrote:
>
>> Something were eating the disk. Disconnected the worker so jobs could be
>> allocated to other nodes. Will look deeper.
>> Filesystem      Size  Used  Avail Use% Mounted on
>> /dev/sda1       485G  485G 96K 100%  /
>>
>>
>> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com> wrote:
>>
>>> I'm on it.
>>>
>>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:
>>>
>>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>>
>>>> Can someone investigate what's going on?
>>>>
>>>

Re: apache-beam-jenkins-15 out of disk

Posted by Yifan Zou <yi...@google.com>.
The problem was because of the large quantity of stale docker images
generated by the Python portable tests and HDFS IT.

Dumping the docker disk usage gives me:

TYPE                TOTAL               ACTIVE              SIZE
     RECLAIMABLE
*Images              1039                356                 424GB
      384.2GB (90%)*
Containers          987                 2                   2.042GB
    2.041GB (99%)
Local Volumes       126                 0                   392.8MB
    392.8MB (100%)

REPOSITORY
          TAG                 IMAGE ID            CREATED             SIZE
               SHARED SIZE         UNIQUE SIZE         CONTAINERS
jenkins-docker-apache.bintray.io/beam/python3
 latest              ff1b949f4442        22 hours ago        1.639GB
      922.3MB                  716.9MB             0
jenkins-docker-apache.bintray.io/beam/python
 latest              1dda7b9d9748        22 hours ago        1.624GB
      913.7MB               710.3MB             0
<none>
                 <none>              05458187a0e3        22 hours ago
 732.9MB             625.1MB            107.8MB             4
<none>
                 <none>              896f35dd685f        23 hours ago
 1.639GB             922.3MB               716.9MB             0
<none>
                 <none>              db4d24ca9f2b        23 hours ago
 1.624GB             913.7MB              710.3MB             0
<none>
                  <none>              547df4d71c31        23 hours ago
   732.9MB             625.1MB             107.8MB             4
<none>
                  <none>              dd7d9582c3e0        23 hours ago
   1.639GB             922.3MB             716.9MB             0
<none>
                  <none>              664aae255239        23 hours ago
   1.624GB             913.7MB             710.3MB             0
<none>
                  <none>              b528fedf9228        23 hours ago
   732.9MB             625.1MB             107.8MB             4
<none>
                  <none>              8e996f22435e        25 hours ago
   1.624GB             913.7MB            710.3MB             0
hdfs_it-jenkins-beam_postcommit_python_verify_pr-818_test    latest
     24b73b3fec06        25 hours ago        1.305GB             965.7MB
           339.5MB             0
<none>
                  <none>              096325fb48de       25 hours ago
 732.9MB             625.1MB            107.8MB              2
jenkins-docker-apache.bintray.io/beam/java
 latest              c36d8ff2945d          25 hours ago        685.6MB
        625.1MB               60.52MB             0
<none>
                  <none>              11c86ebe025f        26 hours ago
   1.639GB             922.3MB              716.9MB             0
<none>
                  <none>              2ecd69c89ec1        26 hours ago
   1.624GB             913.7MB             710.3MB             0
hdfs_it-jenkins-beam_postcommit_python_verify-8590_test        latest
       3d1d589d44fe        2 days ago          1.305GB             965.7MB
             339.5MB             0
hdfs_it-jenkins-beam_postcommit_python_verify_pr-801_test      latest
       d1cc503ebe8e        2 days ago          1.305GB             965.7MB
            339.2MB             0
hdfs_it-jenkins-beam_postcommit_python_verify-8577_test        latest
       8582c6ca6e15        3 days ago          1.305GB             965.7MB
            339.2MB             0
hdfs_it-jenkins-beam_postcommit_python_verify-8576_test        latest
       4591e0948170        3 days ago          1.305GB             965.7MB
            339.2MB             0
hdfs_it-jenkins-beam_postcommit_python_verify-8575_test        latest
       ab181c49d56e        4 days ago          1.305GB             965.7MB
            339.2MB             0
hdfs_it-jenkins-beam_postcommit_python_verify-8573_test        latest
       2104ba0a6db7        4 days ago          1.305GB             965.7MB
            339.2MB             0
...
<1000+ images>

I removed unused the images and the beam15 is back now.

Opened https://issues.apache.org/jira/browse/BEAM-7650.
Ankur, I assigned the issue to you. Feel free to reassign it if needed.

Thank you.
Yifan

On Thu, Jun 27, 2019 at 11:29 AM Yifan Zou <yi...@google.com> wrote:

> Something were eating the disk. Disconnected the worker so jobs could be
> allocated to other nodes. Will look deeper.
> Filesystem      Size  Used  Avail Use% Mounted on
> /dev/sda1       485G  485G 96K 100%  /
>
>
> On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com> wrote:
>
>> I'm on it.
>>
>> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:
>>
>>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>>
>>> Can someone investigate what's going on?
>>>
>>

Re: apache-beam-jenkins-15 out of disk

Posted by Yifan Zou <yi...@google.com>.
Something were eating the disk. Disconnected the worker so jobs could be
allocated to other nodes. Will look deeper.
Filesystem      Size  Used  Avail Use% Mounted on
/dev/sda1       485G  485G 96K 100%  /


On Thu, Jun 27, 2019 at 10:54 AM Yifan Zou <yi...@google.com> wrote:

> I'm on it.
>
> On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:
>
>> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>>
>> Can someone investigate what's going on?
>>
>

Re: apache-beam-jenkins-15 out of disk

Posted by Yifan Zou <yi...@google.com>.
I'm on it.

On Thu, Jun 27, 2019 at 10:17 AM Udi Meiri <eh...@google.com> wrote:

> Opened a bug here: https://issues.apache.org/jira/browse/BEAM-7648
>
> Can someone investigate what's going on?
>