You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Maximilian Michels <mx...@apache.org> on 2020/04/29 19:35:49 UTC
Python 3.7 docker container fails to build
Hi,
has anyone noticed the Python 3.7 Docker container fails to build? I
haven't been able to build the Python 3.7 container, neither locally nor
on Jenkins.
I get:
17:48:10 > Task :sdks:python:container:py37:docker
17:49:36 The command '/bin/sh -c pip install -r
/tmp/base_image_requirements.txt && python -c "from
google.protobuf.internal import api_implementation; assert
api_implementation._default_implementation_type == 'cpp'; print
('Verified fast protobuf used.')" && rm -rf /root/.cache/pip' returned a
non-zero code: 1
17:49:36
17:49:36 > Task :sdks:python:container:py37:docker FAILED
Cheers,
Max
Re: Python 3.7 docker container fails to build
Posted by Ahmet Altay <al...@google.com>.
+1 to periodic cleanups of the workers. I do not know what would be a good
frequency daily or a different one. Do we have a jira for this?
On Thu, Apr 30, 2020 at 2:22 PM Udi Meiri <eh...@google.com> wrote:
> I summarized my idea here: https://issues.apache.org/jira/browse/BEAM-9865
>
+1 to this idea as well.
>
>
> On Thu, Apr 30, 2020 at 2:01 PM Maximilian Michels <mx...@apache.org> wrote:
>
>> On 30.04.20 21:48, Hannah Jiang wrote:
>> > --info tag was passed to docker image build commands with PythonDocker
>> > Precommit to capture more logs. Without the tag, errors from
>> > DockerFile step are not printed out to the console.
>>
>> Thanks for the info (pun intended).
>>
>> On 30.04.20 21:48, Hannah Jiang wrote:
>> > Indeed, I can see the no space left on device in the following but
>> > not in the log above:
>> >
>> > --info tag was passed to docker image build commands with PythonDocker
>> > Precommit to capture more logs. Without the tag, errors from DockerFile
>> > step are not printed out to the console.
>> >
>> > On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri <ehudm@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > I checked node 8 and it had over 40GB space available. Does your job
>> > require more than that?
>> >
>> > Long term, I'm thinking we could clean up workspaces for successful
>> > jobs. This should free up additional space (I guess at least 100GB).
>> > https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin
>> > to clean workspaces at job start.
>> >
>> >
>> > On Thu, Apr 30, 2020, 07:33 Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > *It's working again, probably because it's running on a
>> different
>> > machine now.
>> >
>> > Who can check the disk space of the Jenkins hosts?
>> >
>> > Thanks,
>> > Max
>> >
>> > On 30.04.20 11:55, Maximilian Michels wrote:
>> > > Sorry, I meant to include the Jenkins log:
>> > >
>> >
>> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
>> > >
>> > > Thanks for investigating Hannah! Indeed, I can see the no
>> > space left on
>> > > device in the following but not in the log above:
>> > >
>> >
>> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
>> > >
>> > > I'm going to try running the build again. Do you think we
>> > could add more
>> > > storage to our Jenkins hosts or delete old build data?
>> > >
>> > > Thanks,
>> > > Max
>> > >
>> > > On 30.04.20 08:43, Hannah Jiang wrote:
>> > >> Max, I found a link from your PR and noticed below errors.
>> > This would be
>> > >> the true error.
>> > >>
>> > >> *07:57:03* >*Task :sdks:python:container:py37:docker*
>> > >> *07:57:03* [91mERROR: Could not install packages due to an
>> > EnvironmentError: [Errno 28] No space left on device
>> > >> *07:57:03*
>> > >> *07:57:03* [0m
>> > >> *07:57:03* >*Task :sdks:python:container:py35:docker*
>> > >> *07:57:03* [91mERROR: Could not install packages due to an
>> > EnvironmentError: [Errno 28] No space left on device
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang
>> > <hannahjiang@google.com <ma...@google.com>
>> > >> <mailto:hannahjiang@google.com
>> > <ma...@google.com>>> wrote:
>> > >>
>> > >> There is a PythonDocker Precommit test running for PRs
>> > with Python
>> > >> changes. It seems running well.[1]
>> > >> Max, can you please give me a link so I can check more
>> > details? Do
>> > >> other images with different Python versions fail as well?
>> > >>
>> > >>
>> > 1.
>> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
>> > >>
>> > >>
>> > >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay
>> > <altay@google.com <ma...@google.com>
>> > >> <mailto:altay@google.com <ma...@google.com>>>
>> wrote:
>> > >>
>> > >> +Valentyn Tymofieiev <mailto:valentyn@google.com
>> > <ma...@google.com>> +Hannah Jiang
>> > >> <mailto:hannahjiang@google.com
>> > <ma...@google.com>> -- in case they have relevant
>> > >> information.
>> > >>
>> > >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
>> > >> <mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>> > >>
>> > >> Hi,
>> > >>
>> > >> has anyone noticed the Python 3.7 Docker
>> > container fails to
>> > >> build? I
>> > >> haven't been able to build the Python 3.7
>> > container, neither
>> > >> locally nor
>> > >> on Jenkins.
>> > >>
>> > >> I get:
>> > >>
>> > >> 17:48:10 > Task
>> :sdks:python:container:py37:docker
>> > >> 17:49:36 The command '/bin/sh -c pip install -r
>> > >> /tmp/base_image_requirements.txt && python -c
>> "from
>> > >> google.protobuf.internal import
>> > api_implementation; assert
>> > >> api_implementation._default_implementation_type
>> > == 'cpp'; print
>> > >> ('Verified fast protobuf used.')" && rm -rf
>> > >> /root/.cache/pip' returned a
>> > >> non-zero code: 1
>> > >> 17:49:36
>> > >> 17:49:36 > Task
>> > :sdks:python:container:py37:docker FAILED
>> > >>
>> > >>
>> > >> Cheers,
>> > >> Max
>> > >>
>> >
>>
>
Re: Python 3.7 docker container fails to build
Posted by Udi Meiri <eh...@google.com>.
I summarized my idea here: https://issues.apache.org/jira/browse/BEAM-9865
On Thu, Apr 30, 2020 at 2:01 PM Maximilian Michels <mx...@apache.org> wrote:
> On 30.04.20 21:48, Hannah Jiang wrote:
> > --info tag was passed to docker image build commands with PythonDocker
> > Precommit to capture more logs. Without the tag, errors from
> > DockerFile step are not printed out to the console.
>
> Thanks for the info (pun intended).
>
> On 30.04.20 21:48, Hannah Jiang wrote:
> > Indeed, I can see the no space left on device in the following but
> > not in the log above:
> >
> > --info tag was passed to docker image build commands with PythonDocker
> > Precommit to capture more logs. Without the tag, errors from DockerFile
> > step are not printed out to the console.
> >
> > On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri <ehudm@google.com
> > <ma...@google.com>> wrote:
> >
> > I checked node 8 and it had over 40GB space available. Does your job
> > require more than that?
> >
> > Long term, I'm thinking we could clean up workspaces for successful
> > jobs. This should free up additional space (I guess at least 100GB).
> > https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin
> > to clean workspaces at job start.
> >
> >
> > On Thu, Apr 30, 2020, 07:33 Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > *It's working again, probably because it's running on a different
> > machine now.
> >
> > Who can check the disk space of the Jenkins hosts?
> >
> > Thanks,
> > Max
> >
> > On 30.04.20 11:55, Maximilian Michels wrote:
> > > Sorry, I meant to include the Jenkins log:
> > >
> >
> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
> > >
> > > Thanks for investigating Hannah! Indeed, I can see the no
> > space left on
> > > device in the following but not in the log above:
> > >
> >
> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
> > >
> > > I'm going to try running the build again. Do you think we
> > could add more
> > > storage to our Jenkins hosts or delete old build data?
> > >
> > > Thanks,
> > > Max
> > >
> > > On 30.04.20 08:43, Hannah Jiang wrote:
> > >> Max, I found a link from your PR and noticed below errors.
> > This would be
> > >> the true error.
> > >>
> > >> *07:57:03* >*Task :sdks:python:container:py37:docker*
> > >> *07:57:03* [91mERROR: Could not install packages due to an
> > EnvironmentError: [Errno 28] No space left on device
> > >> *07:57:03*
> > >> *07:57:03* [0m
> > >> *07:57:03* >*Task :sdks:python:container:py35:docker*
> > >> *07:57:03* [91mERROR: Could not install packages due to an
> > EnvironmentError: [Errno 28] No space left on device
> > >>
> > >>
> > >>
> > >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang
> > <hannahjiang@google.com <ma...@google.com>
> > >> <mailto:hannahjiang@google.com
> > <ma...@google.com>>> wrote:
> > >>
> > >> There is a PythonDocker Precommit test running for PRs
> > with Python
> > >> changes. It seems running well.[1]
> > >> Max, can you please give me a link so I can check more
> > details? Do
> > >> other images with different Python versions fail as well?
> > >>
> > >>
> > 1.
> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
> > >>
> > >>
> > >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay
> > <altay@google.com <ma...@google.com>
> > >> <mailto:altay@google.com <ma...@google.com>>>
> wrote:
> > >>
> > >> +Valentyn Tymofieiev <mailto:valentyn@google.com
> > <ma...@google.com>> +Hannah Jiang
> > >> <mailto:hannahjiang@google.com
> > <ma...@google.com>> -- in case they have relevant
> > >> information.
> > >>
> > >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
> > >> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >>
> > >> Hi,
> > >>
> > >> has anyone noticed the Python 3.7 Docker
> > container fails to
> > >> build? I
> > >> haven't been able to build the Python 3.7
> > container, neither
> > >> locally nor
> > >> on Jenkins.
> > >>
> > >> I get:
> > >>
> > >> 17:48:10 > Task :sdks:python:container:py37:docker
> > >> 17:49:36 The command '/bin/sh -c pip install -r
> > >> /tmp/base_image_requirements.txt && python -c
> "from
> > >> google.protobuf.internal import
> > api_implementation; assert
> > >> api_implementation._default_implementation_type
> > == 'cpp'; print
> > >> ('Verified fast protobuf used.')" && rm -rf
> > >> /root/.cache/pip' returned a
> > >> non-zero code: 1
> > >> 17:49:36
> > >> 17:49:36 > Task
> > :sdks:python:container:py37:docker FAILED
> > >>
> > >>
> > >> Cheers,
> > >> Max
> > >>
> >
>
Re: Python 3.7 docker container fails to build
Posted by Maximilian Michels <mx...@apache.org>.
On 30.04.20 21:48, Hannah Jiang wrote:
> --info tag was passed to docker image build commands with PythonDocker
> Precommit to capture more logs. Without the tag, errors from
> DockerFile step are not printed out to the console.
Thanks for the info (pun intended).
On 30.04.20 21:48, Hannah Jiang wrote:
> Indeed, I can see the no space left on device in the following but
> not in the log above:
>
> --info tag was passed to docker image build commands with PythonDocker
> Precommit to capture more logs. Without the tag, errors from DockerFile
> step are not printed out to the console.
>
> On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri <ehudm@google.com
> <ma...@google.com>> wrote:
>
> I checked node 8 and it had over 40GB space available. Does your job
> require more than that?
>
> Long term, I'm thinking we could clean up workspaces for successful
> jobs. This should free up additional space (I guess at least 100GB).
> https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin
> to clean workspaces at job start.
>
>
> On Thu, Apr 30, 2020, 07:33 Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> *It's working again, probably because it's running on a different
> machine now.
>
> Who can check the disk space of the Jenkins hosts?
>
> Thanks,
> Max
>
> On 30.04.20 11:55, Maximilian Michels wrote:
> > Sorry, I meant to include the Jenkins log:
> >
> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
> >
> > Thanks for investigating Hannah! Indeed, I can see the no
> space left on
> > device in the following but not in the log above:
> >
> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
> >
> > I'm going to try running the build again. Do you think we
> could add more
> > storage to our Jenkins hosts or delete old build data?
> >
> > Thanks,
> > Max
> >
> > On 30.04.20 08:43, Hannah Jiang wrote:
> >> Max, I found a link from your PR and noticed below errors.
> This would be
> >> the true error.
> >>
> >> *07:57:03* >*Task :sdks:python:container:py37:docker*
> >> *07:57:03* [91mERROR: Could not install packages due to an
> EnvironmentError: [Errno 28] No space left on device
> >> *07:57:03*
> >> *07:57:03* [0m
> >> *07:57:03* >*Task :sdks:python:container:py35:docker*
> >> *07:57:03* [91mERROR: Could not install packages due to an
> EnvironmentError: [Errno 28] No space left on device
> >>
> >>
> >>
> >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang
> <hannahjiang@google.com <ma...@google.com>
> >> <mailto:hannahjiang@google.com
> <ma...@google.com>>> wrote:
> >>
> >> There is a PythonDocker Precommit test running for PRs
> with Python
> >> changes. It seems running well.[1]
> >> Max, can you please give me a link so I can check more
> details? Do
> >> other images with different Python versions fail as well?
> >>
> >>
> 1. https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
> >>
> >>
> >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay
> <altay@google.com <ma...@google.com>
> >> <mailto:altay@google.com <ma...@google.com>>> wrote:
> >>
> >> +Valentyn Tymofieiev <mailto:valentyn@google.com
> <ma...@google.com>> +Hannah Jiang
> >> <mailto:hannahjiang@google.com
> <ma...@google.com>> -- in case they have relevant
> >> information.
> >>
> >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
> >> <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >>
> >> Hi,
> >>
> >> has anyone noticed the Python 3.7 Docker
> container fails to
> >> build? I
> >> haven't been able to build the Python 3.7
> container, neither
> >> locally nor
> >> on Jenkins.
> >>
> >> I get:
> >>
> >> 17:48:10 > Task :sdks:python:container:py37:docker
> >> 17:49:36 The command '/bin/sh -c pip install -r
> >> /tmp/base_image_requirements.txt && python -c "from
> >> google.protobuf.internal import
> api_implementation; assert
> >> api_implementation._default_implementation_type
> == 'cpp'; print
> >> ('Verified fast protobuf used.')" && rm -rf
> >> /root/.cache/pip' returned a
> >> non-zero code: 1
> >> 17:49:36
> >> 17:49:36 > Task
> :sdks:python:container:py37:docker FAILED
> >>
> >>
> >> Cheers,
> >> Max
> >>
>
Re: Python 3.7 docker container fails to build
Posted by Maximilian Michels <mx...@apache.org>.
Is the issue that the workspace grows over time? Couldn't we delete it
daily to ensure it does not grow too much? Always deleting it on
successful runs may be too costly because we have to recreate the
workspace every time.
Logs are stored separately. I suppose they could also add up over time.
On 30.04.20 21:48, Hannah Jiang wrote:
> Indeed, I can see the no space left on device in the following but
> not in the log above:
>
> --info tag was passed to docker image build commands with PythonDocker
> Precommit to capture more logs. Without the tag, errors from DockerFile
> step are not printed out to the console.
>
> On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri <ehudm@google.com
> <ma...@google.com>> wrote:
>
> I checked node 8 and it had over 40GB space available. Does your job
> require more than that?
>
> Long term, I'm thinking we could clean up workspaces for successful
> jobs. This should free up additional space (I guess at least 100GB).
> https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin
> to clean workspaces at job start.
>
>
> On Thu, Apr 30, 2020, 07:33 Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> *It's working again, probably because it's running on a different
> machine now.
>
> Who can check the disk space of the Jenkins hosts?
>
> Thanks,
> Max
>
> On 30.04.20 11:55, Maximilian Michels wrote:
> > Sorry, I meant to include the Jenkins log:
> >
> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
> >
> > Thanks for investigating Hannah! Indeed, I can see the no
> space left on
> > device in the following but not in the log above:
> >
> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
> >
> > I'm going to try running the build again. Do you think we
> could add more
> > storage to our Jenkins hosts or delete old build data?
> >
> > Thanks,
> > Max
> >
> > On 30.04.20 08:43, Hannah Jiang wrote:
> >> Max, I found a link from your PR and noticed below errors.
> This would be
> >> the true error.
> >>
> >> *07:57:03* >*Task :sdks:python:container:py37:docker*
> >> *07:57:03* [91mERROR: Could not install packages due to an
> EnvironmentError: [Errno 28] No space left on device
> >> *07:57:03*
> >> *07:57:03* [0m
> >> *07:57:03* >*Task :sdks:python:container:py35:docker*
> >> *07:57:03* [91mERROR: Could not install packages due to an
> EnvironmentError: [Errno 28] No space left on device
> >>
> >>
> >>
> >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang
> <hannahjiang@google.com <ma...@google.com>
> >> <mailto:hannahjiang@google.com
> <ma...@google.com>>> wrote:
> >>
> >> There is a PythonDocker Precommit test running for PRs
> with Python
> >> changes. It seems running well.[1]
> >> Max, can you please give me a link so I can check more
> details? Do
> >> other images with different Python versions fail as well?
> >>
> >>
> 1. https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
> >>
> >>
> >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay
> <altay@google.com <ma...@google.com>
> >> <mailto:altay@google.com <ma...@google.com>>> wrote:
> >>
> >> +Valentyn Tymofieiev <mailto:valentyn@google.com
> <ma...@google.com>> +Hannah Jiang
> >> <mailto:hannahjiang@google.com
> <ma...@google.com>> -- in case they have relevant
> >> information.
> >>
> >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
> >> <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >>
> >> Hi,
> >>
> >> has anyone noticed the Python 3.7 Docker
> container fails to
> >> build? I
> >> haven't been able to build the Python 3.7
> container, neither
> >> locally nor
> >> on Jenkins.
> >>
> >> I get:
> >>
> >> 17:48:10 > Task :sdks:python:container:py37:docker
> >> 17:49:36 The command '/bin/sh -c pip install -r
> >> /tmp/base_image_requirements.txt && python -c "from
> >> google.protobuf.internal import
> api_implementation; assert
> >> api_implementation._default_implementation_type
> == 'cpp'; print
> >> ('Verified fast protobuf used.')" && rm -rf
> >> /root/.cache/pip' returned a
> >> non-zero code: 1
> >> 17:49:36
> >> 17:49:36 > Task
> :sdks:python:container:py37:docker FAILED
> >>
> >>
> >> Cheers,
> >> Max
> >>
>
Re: Python 3.7 docker container fails to build
Posted by Hannah Jiang <ha...@google.com>.
>
> Indeed, I can see the no space left on device in the following but not in
> the log above:
--info tag was passed to docker image build commands with PythonDocker
Precommit to capture more logs. Without the tag, errors from DockerFile
step are not printed out to the console.
On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri <eh...@google.com> wrote:
> I checked node 8 and it had over 40GB space available. Does your job
> require more than that?
>
> Long term, I'm thinking we could clean up workspaces for successful jobs.
> This should free up additional space (I guess at least 100GB).
> https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin to
> clean workspaces at job start.
>
>
> On Thu, Apr 30, 2020, 07:33 Maximilian Michels <mx...@apache.org> wrote:
>
>> *It's working again, probably because it's running on a different
>> machine now.
>>
>> Who can check the disk space of the Jenkins hosts?
>>
>> Thanks,
>> Max
>>
>> On 30.04.20 11:55, Maximilian Michels wrote:
>> > Sorry, I meant to include the Jenkins log:
>> >
>> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
>> >
>> > Thanks for investigating Hannah! Indeed, I can see the no space left on
>> > device in the following but not in the log above:
>> >
>> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
>> >
>> > I'm going to try running the build again. Do you think we could add more
>> > storage to our Jenkins hosts or delete old build data?
>> >
>> > Thanks,
>> > Max
>> >
>> > On 30.04.20 08:43, Hannah Jiang wrote:
>> >> Max, I found a link from your PR and noticed below errors. This would
>> be
>> >> the true error.
>> >>
>> >> *07:57:03* >*Task :sdks:python:container:py37:docker*
>> >> *07:57:03* [91mERROR: Could not install packages due to an
>> EnvironmentError: [Errno 28] No space left on device
>> >> *07:57:03*
>> >> *07:57:03* [0m
>> >> *07:57:03* >*Task :sdks:python:container:py35:docker*
>> >> *07:57:03* [91mERROR: Could not install packages due to an
>> EnvironmentError: [Errno 28] No space left on device
>> >>
>> >>
>> >>
>> >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang <hannahjiang@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> There is a PythonDocker Precommit test running for PRs with Python
>> >> changes. It seems running well.[1]
>> >> Max, can you please give me a link so I can check more details? Do
>> >> other images with different Python versions fail as well?
>> >>
>> >> 1.
>> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
>> >>
>> >>
>> >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay <altay@google.com
>> >> <ma...@google.com>> wrote:
>> >>
>> >> +Valentyn Tymofieiev <ma...@google.com> +Hannah
>> Jiang
>> >> <ma...@google.com> -- in case they have relevant
>> >> information.
>> >>
>> >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
>> >> <mxm@apache.org <ma...@apache.org>> wrote:
>> >>
>> >> Hi,
>> >>
>> >> has anyone noticed the Python 3.7 Docker container fails to
>> >> build? I
>> >> haven't been able to build the Python 3.7 container,
>> neither
>> >> locally nor
>> >> on Jenkins.
>> >>
>> >> I get:
>> >>
>> >> 17:48:10 > Task :sdks:python:container:py37:docker
>> >> 17:49:36 The command '/bin/sh -c pip install -r
>> >> /tmp/base_image_requirements.txt && python -c "from
>> >> google.protobuf.internal import api_implementation; assert
>> >> api_implementation._default_implementation_type == 'cpp';
>> print
>> >> ('Verified fast protobuf used.')" && rm -rf
>> >> /root/.cache/pip' returned a
>> >> non-zero code: 1
>> >> 17:49:36
>> >> 17:49:36 > Task :sdks:python:container:py37:docker FAILED
>> >>
>> >>
>> >> Cheers,
>> >> Max
>> >>
>>
>
Re: Python 3.7 docker container fails to build
Posted by Udi Meiri <eh...@google.com>.
I checked node 8 and it had over 40GB space available. Does your job
require more than that?
Long term, I'm thinking we could clean up workspaces for successful jobs.
This should free up additional space (I guess at least 100GB).
https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin to
clean workspaces at job start.
On Thu, Apr 30, 2020, 07:33 Maximilian Michels <mx...@apache.org> wrote:
> *It's working again, probably because it's running on a different
> machine now.
>
> Who can check the disk space of the Jenkins hosts?
>
> Thanks,
> Max
>
> On 30.04.20 11:55, Maximilian Michels wrote:
> > Sorry, I meant to include the Jenkins log:
> >
> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
> >
> > Thanks for investigating Hannah! Indeed, I can see the no space left on
> > device in the following but not in the log above:
> >
> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
> >
> > I'm going to try running the build again. Do you think we could add more
> > storage to our Jenkins hosts or delete old build data?
> >
> > Thanks,
> > Max
> >
> > On 30.04.20 08:43, Hannah Jiang wrote:
> >> Max, I found a link from your PR and noticed below errors. This would be
> >> the true error.
> >>
> >> *07:57:03* >*Task :sdks:python:container:py37:docker*
> >> *07:57:03* [91mERROR: Could not install packages due to an
> EnvironmentError: [Errno 28] No space left on device
> >> *07:57:03*
> >> *07:57:03* [0m
> >> *07:57:03* >*Task :sdks:python:container:py35:docker*
> >> *07:57:03* [91mERROR: Could not install packages due to an
> EnvironmentError: [Errno 28] No space left on device
> >>
> >>
> >>
> >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang <hannahjiang@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> There is a PythonDocker Precommit test running for PRs with Python
> >> changes. It seems running well.[1]
> >> Max, can you please give me a link so I can check more details? Do
> >> other images with different Python versions fail as well?
> >>
> >> 1.
> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
> >>
> >>
> >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay <altay@google.com
> >> <ma...@google.com>> wrote:
> >>
> >> +Valentyn Tymofieiev <ma...@google.com> +Hannah Jiang
> >> <ma...@google.com> -- in case they have relevant
> >> information.
> >>
> >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
> >> <mxm@apache.org <ma...@apache.org>> wrote:
> >>
> >> Hi,
> >>
> >> has anyone noticed the Python 3.7 Docker container fails to
> >> build? I
> >> haven't been able to build the Python 3.7 container, neither
> >> locally nor
> >> on Jenkins.
> >>
> >> I get:
> >>
> >> 17:48:10 > Task :sdks:python:container:py37:docker
> >> 17:49:36 The command '/bin/sh -c pip install -r
> >> /tmp/base_image_requirements.txt && python -c "from
> >> google.protobuf.internal import api_implementation; assert
> >> api_implementation._default_implementation_type == 'cpp';
> print
> >> ('Verified fast protobuf used.')" && rm -rf
> >> /root/.cache/pip' returned a
> >> non-zero code: 1
> >> 17:49:36
> >> 17:49:36 > Task :sdks:python:container:py37:docker FAILED
> >>
> >>
> >> Cheers,
> >> Max
> >>
>
Re: Python 3.7 docker container fails to build
Posted by Maximilian Michels <mx...@apache.org>.
*It's working again, probably because it's running on a different
machine now.
Who can check the disk space of the Jenkins hosts?
Thanks,
Max
On 30.04.20 11:55, Maximilian Michels wrote:
> Sorry, I meant to include the Jenkins log:
> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
>
> Thanks for investigating Hannah! Indeed, I can see the no space left on
> device in the following but not in the log above:
> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
>
> I'm going to try running the build again. Do you think we could add more
> storage to our Jenkins hosts or delete old build data?
>
> Thanks,
> Max
>
> On 30.04.20 08:43, Hannah Jiang wrote:
>> Max, I found a link from your PR and noticed below errors. This would be
>> the true error.
>>
>> *07:57:03* >*Task :sdks:python:container:py37:docker*
>> *07:57:03* [91mERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
>> *07:57:03*
>> *07:57:03* [0m
>> *07:57:03* >*Task :sdks:python:container:py35:docker*
>> *07:57:03* [91mERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
>>
>>
>>
>> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang <hannahjiang@google.com
>> <ma...@google.com>> wrote:
>>
>> There is a PythonDocker Precommit test running for PRs with Python
>> changes. It seems running well.[1]
>> Max, can you please give me a link so I can check more details? Do
>> other images with different Python versions fail as well?
>>
>> 1. https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
>>
>>
>> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay <altay@google.com
>> <ma...@google.com>> wrote:
>>
>> +Valentyn Tymofieiev <ma...@google.com> +Hannah Jiang
>> <ma...@google.com> -- in case they have relevant
>> information.
>>
>> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
>> <mxm@apache.org <ma...@apache.org>> wrote:
>>
>> Hi,
>>
>> has anyone noticed the Python 3.7 Docker container fails to
>> build? I
>> haven't been able to build the Python 3.7 container, neither
>> locally nor
>> on Jenkins.
>>
>> I get:
>>
>> 17:48:10 > Task :sdks:python:container:py37:docker
>> 17:49:36 The command '/bin/sh -c pip install -r
>> /tmp/base_image_requirements.txt && python -c "from
>> google.protobuf.internal import api_implementation; assert
>> api_implementation._default_implementation_type == 'cpp'; print
>> ('Verified fast protobuf used.')" && rm -rf
>> /root/.cache/pip' returned a
>> non-zero code: 1
>> 17:49:36
>> 17:49:36 > Task :sdks:python:container:py37:docker FAILED
>>
>>
>> Cheers,
>> Max
>>
Re: Python 3.7 docker container fails to build
Posted by Maximilian Michels <mx...@apache.org>.
Sorry, I meant to include the Jenkins log:
https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console
Thanks for investigating Hannah! Indeed, I can see the no space left on
device in the following but not in the log above:
https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console
I'm going to try running the build again. Do you think we could add more
storage to our Jenkins hosts or delete old build data?
Thanks,
Max
On 30.04.20 08:43, Hannah Jiang wrote:
> Max, I found a link from your PR and noticed below errors. This would be
> the true error.
>
> *07:57:03* >*Task :sdks:python:container:py37:docker*
> *07:57:03* [91mERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
> *07:57:03*
> *07:57:03* [0m
> *07:57:03* >*Task :sdks:python:container:py35:docker*
> *07:57:03* [91mERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
>
>
>
> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang <hannahjiang@google.com
> <ma...@google.com>> wrote:
>
> There is a PythonDocker Precommit test running for PRs with Python
> changes. It seems running well.[1]
> Max, can you please give me a link so I can check more details? Do
> other images with different Python versions fail as well?
>
> 1. https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
>
>
> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay <altay@google.com
> <ma...@google.com>> wrote:
>
> +Valentyn Tymofieiev <ma...@google.com> +Hannah Jiang
> <ma...@google.com> -- in case they have relevant
> information.
>
> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>> wrote:
>
> Hi,
>
> has anyone noticed the Python 3.7 Docker container fails to
> build? I
> haven't been able to build the Python 3.7 container, neither
> locally nor
> on Jenkins.
>
> I get:
>
> 17:48:10 > Task :sdks:python:container:py37:docker
> 17:49:36 The command '/bin/sh -c pip install -r
> /tmp/base_image_requirements.txt && python -c "from
> google.protobuf.internal import api_implementation; assert
> api_implementation._default_implementation_type == 'cpp'; print
> ('Verified fast protobuf used.')" && rm -rf
> /root/.cache/pip' returned a
> non-zero code: 1
> 17:49:36
> 17:49:36 > Task :sdks:python:container:py37:docker FAILED
>
>
> Cheers,
> Max
>
Re: Python 3.7 docker container fails to build
Posted by Hannah Jiang <ha...@google.com>.
Max, I found a link from your PR and noticed below errors. This would be
the true error.
*07:57:03* >* Task :sdks:python:container:py37:docker**07:57:03*
[91mERROR: Could not install packages due to an EnvironmentError:
[Errno 28] No space left on device*07:57:03* *07:57:03* [0m*07:57:03*
>* Task :sdks:python:container:py35:docker**07:57:03* [91mERROR:
Could not install packages due to an EnvironmentError: [Errno 28] No
space left on device
On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang <ha...@google.com> wrote:
> There is a PythonDocker Precommit test running for PRs with Python
> changes. It seems running well.[1]
> Max, can you please give me a link so I can check more details? Do other
> images with different Python versions fail as well?
>
> 1. https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
>
>
> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay <al...@google.com> wrote:
>
>> +Valentyn Tymofieiev <va...@google.com> +Hannah Jiang
>> <ha...@google.com> -- in case they have relevant information.
>>
>> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> Hi,
>>>
>>> has anyone noticed the Python 3.7 Docker container fails to build? I
>>> haven't been able to build the Python 3.7 container, neither locally nor
>>> on Jenkins.
>>>
>>> I get:
>>>
>>> 17:48:10 > Task :sdks:python:container:py37:docker
>>> 17:49:36 The command '/bin/sh -c pip install -r
>>> /tmp/base_image_requirements.txt && python -c "from
>>> google.protobuf.internal import api_implementation; assert
>>> api_implementation._default_implementation_type == 'cpp'; print
>>> ('Verified fast protobuf used.')" && rm -rf /root/.cache/pip' returned a
>>> non-zero code: 1
>>> 17:49:36
>>> 17:49:36 > Task :sdks:python:container:py37:docker FAILED
>>>
>>>
>>> Cheers,
>>> Max
>>>
>>>
Re: Python 3.7 docker container fails to build
Posted by Hannah Jiang <ha...@google.com>.
There is a PythonDocker Precommit test running for PRs with Python changes.
It seems running well.[1]
Max, can you please give me a link so I can check more details? Do other
images with different Python versions fail as well?
1. https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/
On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay <al...@google.com> wrote:
> +Valentyn Tymofieiev <va...@google.com> +Hannah Jiang
> <ha...@google.com> -- in case they have relevant information.
>
> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels <mx...@apache.org>
> wrote:
>
>> Hi,
>>
>> has anyone noticed the Python 3.7 Docker container fails to build? I
>> haven't been able to build the Python 3.7 container, neither locally nor
>> on Jenkins.
>>
>> I get:
>>
>> 17:48:10 > Task :sdks:python:container:py37:docker
>> 17:49:36 The command '/bin/sh -c pip install -r
>> /tmp/base_image_requirements.txt && python -c "from
>> google.protobuf.internal import api_implementation; assert
>> api_implementation._default_implementation_type == 'cpp'; print
>> ('Verified fast protobuf used.')" && rm -rf /root/.cache/pip' returned a
>> non-zero code: 1
>> 17:49:36
>> 17:49:36 > Task :sdks:python:container:py37:docker FAILED
>>
>>
>> Cheers,
>> Max
>>
>>
Re: Python 3.7 docker container fails to build
Posted by Ahmet Altay <al...@google.com>.
+Valentyn Tymofieiev <va...@google.com> +Hannah Jiang
<ha...@google.com> -- in case they have relevant information.
On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels <mx...@apache.org> wrote:
> Hi,
>
> has anyone noticed the Python 3.7 Docker container fails to build? I
> haven't been able to build the Python 3.7 container, neither locally nor
> on Jenkins.
>
> I get:
>
> 17:48:10 > Task :sdks:python:container:py37:docker
> 17:49:36 The command '/bin/sh -c pip install -r
> /tmp/base_image_requirements.txt && python -c "from
> google.protobuf.internal import api_implementation; assert
> api_implementation._default_implementation_type == 'cpp'; print
> ('Verified fast protobuf used.')" && rm -rf /root/.cache/pip' returned a
> non-zero code: 1
> 17:49:36
> 17:49:36 > Task :sdks:python:container:py37:docker FAILED
>
>
> Cheers,
> Max
>
>