You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "hellojukay (via GitHub)" <gi...@apache.org> on 2023/07/18 05:47:19 UTC
[GitHub] [apisix] hellojukay opened a new issue, #9857: help request: docker pull huge image failed
hellojukay opened a new issue, #9857:
URL: https://github.com/apache/apisix/issues/9857
### Description
When use apisix as gateway , when network slow (5M/s) and docker pull huge image ( image size more than 1GB) always failed
```
$ docker pull docker.xyz.com/jadt/iii:v3
v3: Pulling from jadt/iii
fb668870d8a7: Pull complete
4542784317be: Pull complete
e0bec5df5af5: Pull complete
4053f75740ab: Pull complete
57e09105cdfd: Pull complete
606761d225e5: Pull complete
69473a703fb4: Pull complete
a08ab4e0594b: Pull complete
4cd507bccac2: Pull complete
fa92f16621a4: Downloading [==================================================>] 1.412GB/1.412GB
6dc2b05bd224: Download complete
bdd5f6455bdd: Downloading [==================================================>] 1.146GB/1.146GB
48d915a4e6f6: Download complete
2a70c506252e: Downloading [==================================================>] 3.088GB/3.088GB
4f4fb700ef54: Download complete
bfe9228f18ac: Download complete
515667f151fa: Download complete
faa704319aa2: Downloading [==================================================>] 2.13GB/2.13GB
unexpected EOF
```
But when then gateway switch to nginx ingress , docker pull success (Also in case of slow network)
# Nginx ingress config
```
$ kubectl describe ingress docker-platform-artifactory-ha -n jfrog
Name: docker-platform-artifactory-ha
Labels: app=artifactory-ha
app.kubernetes.io/managed-by=Helm
chart=artifactory-ha-107.24.3
heritage=Helm
release=jfrog-platform
Namespace: jfrog
Address: 10.100.0.5
Ingress Class: <none>
Default backend: 10.100.25.63:8082,10.100.25.72:8082
TLS:
xyz.com terminates
Rules:
Host Path Backends
---- ---- --------
docker.xyz.com
/ jfrog-platform-1-artifactory-ha:8082 (10.100.25.63:8082,10.100.25.72:8082)
Annotations: field.cattle.io/publicEndpoints:
[{"addresses":["10.100.0.5"],"port":80,"protocol":"HTTP","serviceName":"jfrog:jfrog-platform-1-artifactory-ha","ingressName":"jfrog:docker...
kubernetes.io/ingress.class: devops-gateway
meta.helm.sh/release-name: jfrog-platform
meta.helm.sh/release-namespace: jfrog
nginx.ingress.kubernetes.io/configuration-snippet:
rewrite ^/v2/docker-local/(.*) /artifactory/api/docker/docker-local/v2/$1 break;
rewrite ^/(v2)/(.*) /artifactory/api/docker/docker/$1/$2;
nginx.ingress.kubernetes.io/proxy-body-size: 20480m
nginx.ingress.kubernetes.io/proxy-connect-timeout: 60
nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 0
nginx.ingress.kubernetes.io/proxy-read-timeout: 300
nginx.ingress.kubernetes.io/proxy-send-timeout: 300
Events: <none>
```
### Environment
- APISIX version (run `apisix version`):
```
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua version
2.15.0
```
- Operating system (run `uname -a`):
```
Linux apisix-5d6c8565cd-dnlll 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 Linux
```
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
```
bash-5.1# openresty -V
nginx version: openresty/1.21.4.1
built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424)
built with OpenSSL 1.1.1g 21 Apr 2020
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.1.1 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-ap
i/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../apisix-nginx-module-1.9.0 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/stream --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/meta --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../wasm-nginx-module-0.6.2 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../lua-var-nginx-module-v0.5.3 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_
smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
```
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
```
bash-5.1# luarocks --version
/usr/local/bin/luarocks 3.8.0
LuaRocks main command-line interface
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1655330202
@hellojukay Is the problem solved? If I don't receive your reply within 14 days, I will close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed
Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1639537424
# docker info
```
$ docker info
Client:
Version: 24.0.4
Context: orbstack
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.1
Path: /Users/licong/.docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.19.1
Path: /Users/licong/.docker/cli-plugins/docker-compose
dev: Docker Dev Environments (Docker Inc.)
Version: v0.1.0
Path: /usr/local/lib/docker/cli-plugins/docker-dev
extension: Manages Docker extensions (Docker Inc.)
Version: v0.2.18
Path: /usr/local/lib/docker/cli-plugins/docker-extension
sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
Version: 0.6.0
Path: /usr/local/lib/docker/cli-plugins/docker-sbom
scan: Docker Scan (Docker Inc.)
Version: v0.25.0
Path: /usr/local/lib/docker/cli-plugins/docker-scan
scout: Command line tool for Docker Scout (Docker Inc.)
Version: v0.6.0
Path: /usr/local/lib/docker/cli-plugins/docker-scout
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 2
Server Version: 24.0.4
Storage Driver: overlay2
Backing Filesystem: btrfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version: 0cae528dd6cb557f7201036e9f43420650207b58
runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
init version:
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.3.12-orbstack-00210-ga4f4fae8883e
Operating System: OrbStack
OSType: linux
Architecture: aarch64
CPUs: 8
Total Memory: 5.132GiB
Name: docker
ID: 49b01c08-d321-4ca7-a028-0053410b8142
Docker Root Dir: /var/lib/docker
Debug Mode: false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed
Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641874013
[apisix.log](https://github.com/apache/apisix/files/12094189/apisix.log)
this is the error log (merge by 2 pods. filter only error level , `grep [error]` )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641916585
If you are familiar with nginx, you can directly use this configuration to handle this kind of special upload and download requests.
After configuration, you can check nginx.conf to confirm.
https://apisix.apache.org/docs/apisix/customize-nginx-configuration/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641892441
```
2023-07-18_00-00-00__error.log:2023/07/17 09:30:38 [warn] 48#48: *75095135 an upstream response is buffered to a temporary file /usr/local/apisix/proxy_temp/0/83/0000017830 while reading upstream, client: 10.100.11.44, server: _, request: "GET /v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c HTTP/1.1", upstream: "http://10.100.25.63:8082/artifactory/api/docker/docker/v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c", host: "docker.xyz.com"
```
@leslie-tsang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] AlinsRan closed issue #9857: help request: docker pull huge image failed
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan closed issue #9857: help request: docker pull huge image failed
URL: https://github.com/apache/apisix/issues/9857
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed
Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641900255
@Sn0rt Can't this configuration work?

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed
Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641890199
[error.log](https://github.com/apache/apisix/files/12094296/error.log)
@Sn0rt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed
Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641460765
@Sn0rt It is very difficult to promote the change of the gateway in the production environment. Recently, there have been several failures in the infrastructure. Everyone is very careful when changing the infrastructure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641482669
> @Sn0rt It is very difficult to promote the change of the gateway in the production environment. Recently, there have been several failures in the infrastructure. Everyone is very careful when changing the infrastructure.
In the production environment, can you check the log of the gateway? Is there any log of ERROR level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed
Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641905093
> `2023-07-18_00-00-00__error.log:2023/07/17 09:30:38 [warn] 48#48: *75095135 an upstream response is buffered to a temporary file /usr/local/apisix/proxy_temp/0/83/0000017830 while reading upstream, client: 10.100.11.44, server: _, request: "GET /v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c HTTP/1.1", upstream: "http://10.100.25.63:8082/artifactory/api/docker/docker/v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c", host: "docker.xyz.com"`
>
> 
>
> pls take a look this link https://apisix.apache.org/docs/apisix/plugins/proxy-control/
Should i disable `response buffer` , it not releated to `request buffer` , Am I understanding this correctly? As described in this article https://segmentfault.com/a/1190000021704996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641881106
> [apisix.log](https://github.com/apache/apisix/files/12094189/apisix.log) this is the error log (merge by 2 pods.)
if you can attach warn level log is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed
Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641367164
hi [hellojukay](https://github.com/hellojukay), can you set the log level with debug? and take a look at the error.log file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org