You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/08/02 09:00:32 UTC
[GitHub] marcoabreu opened a new issue #11999: Corrupt docker tar in CI
marcoabreu opened a new issue #11999: Corrupt docker tar in CI
URL: https://github.com/apache/incubator-mxnet/issues/11999
During docker pull, the CI fails to download the docker layers:
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/master/1321/pipeline
```
build.py: 2018-08-02 07:58:49,180 3 out of 3 tries to build the docker image.
build.py: 2018-08-02 07:58:49,180 Running command: 'docker build -f docker/Dockerfile.build.centos7_gpu --build-arg USER_ID=1001 --build-arg GROUP_ID=1001 --cache-from mxnetci/build.centos7_gpu -t mxnetci/build.centos7_gpu docker'
Sending build context to Docker daemon 150kB
Step 1/14 : FROM nvidia/cuda:9.1-cudnn7-devel-centos7
9.1-cudnn7-devel-centos7: Pulling from nvidia/cuda
7dc0dca2b151: Already exists
635f539bce28: Pulling fs layer
b530d160452d: Pulling fs layer
708d351e5650: Pulling fs layer
ebed533092e0: Pulling fs layer
0966b9ded8e5: Pulling fs layer
d883c1c4a803: Pulling fs layer
b103f26d08e3: Pulling fs layer
ebed533092e0: Waiting
d883c1c4a803: Waiting
635f539bce28: Verifying Checksum
b530d160452d: Verifying Checksum
b530d160452d: Download complete
635f539bce28: Pull complete
b530d160452d: Pull complete
708d351e5650: Download complete
708d351e5650: Pull complete
ebed533092e0: Verifying Checksum
ebed533092e0: Download complete
ebed533092e0: Pull complete
0966b9ded8e5: Verifying Checksum
0966b9ded8e5: Download complete
d883c1c4a803: Download complete
b103f26d08e3: Verifying Checksum
b103f26d08e3: Download complete
failed to register layer: Error processing tar file(exit status 1: unpigz: abort: corrupted gzip stream -- crc32 mismatch: <stdin>
):
build.py: 2018-08-02 07:58:58,027 Failed to build docker image
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.5/logging/__init__.py", line 980, in emit
msg = self.format(record)
File "/usr/lib/python3.5/logging/__init__.py", line 830, in format
return fmt.format(record)
File "/usr/lib/python3.5/logging/__init__.py", line 567, in format
record.message = record.getMessage()
File "/usr/lib/python3.5/logging/__init__.py", line 330, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "ci/build.py", line 408, in <module>
File "ci/build.py", line 329, in main
File "ci/build.py", line 117, in build_docker
Message: 'Exception during build of docker image'
Arguments: (CalledProcessError(1, ['docker', 'build', '-f', 'docker/Dockerfile.build.centos7_gpu', '--build-arg', 'USER_ID=1001', '--build-arg', 'GROUP_ID=1001', '--cache-from', 'mxnetci/build.centos7_gpu', '-t', 'mxnetci/build.centos7_gpu', 'docker']),)
build.py: 2018-08-02 07:58:58,028 Failed to build the docker image, aborting...
script returned exit code 1
```
This could be an error on our side, but I rather think that our slaves had a problematic internet connection. If you have a look at the pasted CI run, you will see that there have been other download corruptions:
```
Cloning into '3rdparty/googletest'...
fatal: did not receive expected object 662a8ab6d0910bf16d3b8e673c8152b758d65cd4
fatal: index-pack failed
fatal: clone of 'https://github.com/google/googletest.git' into submodule path '3rdparty/googletest' failed
script returned exit code 128
```
```
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://github.com/apache/incubator-mxnet.git
> git init /home/jenkins_slave/workspace/build-centos7-gpu # timeout=10
Fetching upstream changes from https://github.com/apache/incubator-mxnet.git
> git --version # timeout=10
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress https://github.com/apache/incubator-mxnet.git +refs/heads/master:refs/remotes/origin/master
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https://github.com/apache/incubator-mxnet.git +refs/heads/master:refs/remotes/origin/master" returned status code 128:
stdout:
stderr: remote: Counting objects: 66937, done.
remote: Compressing objects: 4% (1/23)
remote: Compressing objects: 8% (2/23)
remote: Compressing objects: 13% (3/23)
remote: Compressing objects: 17% (4/23)
remote: Compressing objects: 21% (5/23)
remote: Compressing objects: 26% (6/23)
remote: Compressing objects: 30% (7/23)
remote: Compressing objects: 34% (8/23)
remote: Compressing objects: 39% (9/23)
remote: Compressing objects: 43% (10/23)
remote: Compressing objects: 47% (11/23)
remote: Compressing objects: 52% (12/23)
remote: Compressing objects: 56% (13/23)
remote: Compressing objects: 60% (14/23)
remote: Compressing objects: 65% (15/23)
remote: Compressing objects: 69% (16/23)
remote: Compressing objects: 73% (17/23)
remote: Compressing objects: 78% (18/23)
remote: Compressing objects: 82% (19/23)
remote: Compressing objects: 86% (20/23)
remote: Compressing objects: 91% (21/23)
remote: Compressing objects: 95% (22/23)
remote: Compressing objects: 100% (23/23)
remote: Compressing objects: 100% (23/23), done.
Receiving objects: 0% (1/66937)
Receiving objects: 1% (670/66937)
Receiving objects: 2% (1339/66937)
Receiving objects: 3% (2009/66937)
Receiving objects: 4% (2678/66937)
Receiving objects: 5% (3347/66937)
Receiving objects: 6% (4017/66937)
Receiving objects: 7% (4686/66937)
Receiving objects: 8% (5355/66937)
Receiving objects: 9% (6025/66937)
Receiving objects: 10% (6694/66937)
Receiving objects: 11% (7364/66937)
Receiving objects: 12% (8033/66937)
Receiving objects: 13% (8702/66937)
error: RPC failed; curl 56 GnuTLS recv error (-24): Decryption has failed.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:614)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-172-31-13-44.us-west-2.compute.internal/172.31.13.44:33806
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:955)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor602.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy94.execute(Unknown Source)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Error cloning remote repo 'origin'
```
I imagine that we can't do anything here and that it's simply a bad slave.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services