You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Jie Yu (JIRA)" <ji...@apache.org> on 2018/01/03 19:53:00 UTC
[jira] [Comment Edited] (MESOS-8253) Mesos CI docker rmi conflict
[ https://issues.apache.org/jira/browse/MESOS-8253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310193#comment-16310193 ]
Jie Yu edited comment on MESOS-8253 at 1/3/18 7:52 PM:
-------------------------------------------------------
I did some triage on this issue. It looks like that the stdout and stderr are marked as non-blocking. I added a few print in our docker_build.sh script to verify
{noformat}
+ python -c 'import os,sys,fcntl; flags = fcntl.fcntl(sys.stdout, fcntl.F_GETFL); print(flags&os.O_NONBLOCK);'
2048
+ python -c 'import os,sys,fcntl; flags = fcntl.fcntl(sys.stderr, fcntl.F_GETFL); print(flags&os.O_NONBLOCK);'
2048
{noformat}
This explains the EAGAIN received by `docker run`. I couldn't find a way in Jenkins to set the stdout/stderr to be blocking.
A workaround I am testing is to dump stdout/stderr to a file and tail the file in the end to reduce the likelihood of this. But the true fix is to set fds to be blocking.
Related issues:
https://github.com/travis-ci/travis-ci/issues/4704
was (Author: jieyu):
I did some triage on this issue. It looks like that the stdout and stderr are marked as non-blocking. I added a few print in our docker_build.sh script to verify
{noformat}
+ python -c 'import os,sys,fcntl; flags = fcntl.fcntl(sys.stdout, fcntl.F_GETFL); print(flags&os.O_NONBLOCK);'
2048
+ python -c 'import os,sys,fcntl; flags = fcntl.fcntl(sys.stderr, fcntl.F_GETFL); print(flags&os.O_NONBLOCK);'
2048
{noformat}
This explains the EAGAIN received by `docker run`. I couldn't find a way in Jenkins to set the stdout/stderr to be blocking.
A workaround I am testing is to dump stdout/stderr to a file and tail the file in the end to reduce the likelihood of this. But the true fix is to set fds to be blocking.
> Mesos CI docker rmi conflict
> ----------------------------
>
> Key: MESOS-8253
> URL: https://issues.apache.org/jira/browse/MESOS-8253
> Project: Mesos
> Issue Type: Bug
> Components: build, docker
> Reporter: James Peach
>
> We are seeing a lot of docker build jobs failing when they try to clean up there docker images:
> {noformat}
> + docker rmi mesos-1511286604-15916
> Error response from daemon: conflict: unable to remove repository reference "mesos-1511286604-15916" (must force) - container 1aabf0225a43 is using its referenced image 23292073f88f
> Build step 'Execute shell' marked build as failure
> {noformat}
> The full Jenkins log is [here|https://builds.apache.org/job/Mesos-Buildbot/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A14.04,label_exp=(ubuntu)&&(!ubuntu-us1)&&(!ubuntu-eu2)&&(!qnode3)&&(!H23)/4486/console]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)