You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "hellojukay (via GitHub)" <gi...@apache.org> on 2023/07/18 05:47:19 UTC

[GitHub] [apisix] hellojukay opened a new issue, #9857: help request: docker pull huge image failed

hellojukay opened a new issue, #9857:
URL: https://github.com/apache/apisix/issues/9857

   ### Description
   
   When use apisix as gateway , when network slow (5M/s)  and docker pull huge image ( image size more than 1GB) always failed
   ```
   $ docker  pull docker.xyz.com/jadt/iii:v3
   v3: Pulling from jadt/iii
   fb668870d8a7: Pull complete
   4542784317be: Pull complete
   e0bec5df5af5: Pull complete
   4053f75740ab: Pull complete
   57e09105cdfd: Pull complete
   606761d225e5: Pull complete
   69473a703fb4: Pull complete
   a08ab4e0594b: Pull complete
   4cd507bccac2: Pull complete
   fa92f16621a4: Downloading [==================================================>]  1.412GB/1.412GB
   6dc2b05bd224: Download complete
   bdd5f6455bdd: Downloading [==================================================>]  1.146GB/1.146GB
   48d915a4e6f6: Download complete
   2a70c506252e: Downloading [==================================================>]  3.088GB/3.088GB
   4f4fb700ef54: Download complete
   bfe9228f18ac: Download complete
   515667f151fa: Download complete
   faa704319aa2: Downloading [==================================================>]   2.13GB/2.13GB
   unexpected EOF
   ```
   But when then gateway switch to  nginx ingress , docker pull success (Also in case of slow network)
   
   # Nginx ingress config
   ```
   $ kubectl describe ingress docker-platform-artifactory-ha -n jfrog
   Name:             docker-platform-artifactory-ha
   Labels:           app=artifactory-ha
                     app.kubernetes.io/managed-by=Helm
                     chart=artifactory-ha-107.24.3
                     heritage=Helm
                     release=jfrog-platform
   Namespace:        jfrog
   Address:          10.100.0.5
   Ingress Class:    <none>
   Default backend:  10.100.25.63:8082,10.100.25.72:8082
   TLS:
     xyz.com terminates
   Rules:
     Host                Path  Backends
     ----                ----  --------
     docker.xyz.com
                         /   jfrog-platform-1-artifactory-ha:8082 (10.100.25.63:8082,10.100.25.72:8082)
   Annotations:          field.cattle.io/publicEndpoints:
                           [{"addresses":["10.100.0.5"],"port":80,"protocol":"HTTP","serviceName":"jfrog:jfrog-platform-1-artifactory-ha","ingressName":"jfrog:docker...
                         kubernetes.io/ingress.class: devops-gateway
                         meta.helm.sh/release-name: jfrog-platform
                         meta.helm.sh/release-namespace: jfrog
                         nginx.ingress.kubernetes.io/configuration-snippet:
                           rewrite ^/v2/docker-local/(.*) /artifactory/api/docker/docker-local/v2/$1 break;
                           rewrite ^/(v2)/(.*) /artifactory/api/docker/docker/$1/$2;
                         nginx.ingress.kubernetes.io/proxy-body-size: 20480m
                         nginx.ingress.kubernetes.io/proxy-connect-timeout: 60
                         nginx.ingress.kubernetes.io/proxy-max-temp-file-size: 0
                         nginx.ingress.kubernetes.io/proxy-read-timeout: 300
                         nginx.ingress.kubernetes.io/proxy-send-timeout: 300
   Events:               <none>
   ``` 
   
   ### Environment
   
   - APISIX version (run `apisix version`):
   ```
   /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua version
   2.15.0
   ```
   - Operating system (run `uname -a`):
   ```
   Linux apisix-5d6c8565cd-dnlll 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 Linux
   ```
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
   ```
   bash-5.1# openresty -V
   nginx version: openresty/1.21.4.1
   built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424)
   built with OpenSSL 1.1.1g  21 Apr 2020
   TLS SNI support enabled
   configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.1.1 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.1 --add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.21 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.33 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.11 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-ap
 i/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../apisix-nginx-module-1.9.0 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/stream --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../apisix-nginx-module-1.9.0/src/meta --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../wasm-nginx-module-0.6.2 --add-module=/tmp/tmp.M4fAebVwPS/openresty-1.21.4.1/../lua-var-nginx-module-v0.5.3 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_
 smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
   ```
   - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
   - APISIX Dashboard version, if relevant:
   - Plugin runner version, for issues related to plugin runners:
   - LuaRocks version, for installation issues (run `luarocks --version`):
   ```
   bash-5.1# luarocks --version
   /usr/local/bin/luarocks 3.8.0
   LuaRocks main command-line interface
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed

Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1655330202

   @hellojukay Is the problem solved? If I don't receive your reply within 14 days, I will close this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed

Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1639537424

   # docker info
   ```
   $ docker info
   Client:
    Version:    24.0.4
    Context:    orbstack
    Debug Mode: false
    Plugins:
     buildx: Docker Buildx (Docker Inc.)
       Version:  v0.11.1
       Path:     /Users/licong/.docker/cli-plugins/docker-buildx
     compose: Docker Compose (Docker Inc.)
       Version:  v2.19.1
       Path:     /Users/licong/.docker/cli-plugins/docker-compose
     dev: Docker Dev Environments (Docker Inc.)
       Version:  v0.1.0
       Path:     /usr/local/lib/docker/cli-plugins/docker-dev
     extension: Manages Docker extensions (Docker Inc.)
       Version:  v0.2.18
       Path:     /usr/local/lib/docker/cli-plugins/docker-extension
     sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
       Version:  0.6.0
       Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
     scan: Docker Scan (Docker Inc.)
       Version:  v0.25.0
       Path:     /usr/local/lib/docker/cli-plugins/docker-scan
     scout: Command line tool for Docker Scout (Docker Inc.)
       Version:  v0.6.0
       Path:     /usr/local/lib/docker/cli-plugins/docker-scout
   
   Server:
    Containers: 1
     Running: 0
     Paused: 0
     Stopped: 1
    Images: 2
    Server Version: 24.0.4
    Storage Driver: overlay2
     Backing Filesystem: btrfs
     Supports d_type: true
     Using metacopy: false
     Native Overlay Diff: true
     userxattr: false
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Cgroup Version: 2
    Plugins:
     Volume: local
     Network: bridge host ipvlan macvlan null overlay
     Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
    Swarm: inactive
    Runtimes: runc io.containerd.runc.v2
    Default Runtime: runc
    Init Binary: docker-init
    containerd version: 0cae528dd6cb557f7201036e9f43420650207b58
    runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
    init version:
    Security Options:
     seccomp
      Profile: builtin
     cgroupns
    Kernel Version: 6.3.12-orbstack-00210-ga4f4fae8883e
    Operating System: OrbStack
    OSType: linux
    Architecture: aarch64
    CPUs: 8
    Total Memory: 5.132GiB
    Name: docker
    ID: 49b01c08-d321-4ca7-a028-0053410b8142
    Docker Root Dir: /var/lib/docker
    Debug Mode: false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed

Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641874013

   [apisix.log](https://github.com/apache/apisix/files/12094189/apisix.log)
   this is the error log (merge by 2 pods. filter only error level , `grep [error]` )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed

Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641916585

   If you are familiar with nginx, you can directly use this configuration to handle this kind of special upload and download requests.
   After configuration, you can check nginx.conf to confirm.
   
   https://apisix.apache.org/docs/apisix/customize-nginx-configuration/ 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed

Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641892441

   ```
   2023-07-18_00-00-00__error.log:2023/07/17 09:30:38 [warn] 48#48: *75095135 an upstream response is buffered to a temporary file /usr/local/apisix/proxy_temp/0/83/0000017830 while reading upstream, client: 10.100.11.44, server: _, request: "GET /v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c HTTP/1.1", upstream: "http://10.100.25.63:8082/artifactory/api/docker/docker/v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c", host: "docker.xyz.com"
   ```
   
   @leslie-tsang 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] AlinsRan closed issue #9857: help request: docker pull huge image failed

Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan closed issue #9857: help request:  docker pull huge image failed
URL: https://github.com/apache/apisix/issues/9857


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed

Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641900255

   @Sn0rt Can't this configuration work?
   ![image](https://github.com/apache/apisix/assets/10371891/6513619c-b1e7-4e14-913d-015914417779)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed

Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641890199

   [error.log](https://github.com/apache/apisix/files/12094296/error.log)
   @Sn0rt 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed

Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641460765

   @Sn0rt It is very difficult to promote the change of the gateway in the production environment. Recently, there have been several failures in the infrastructure. Everyone is very careful when changing the infrastructure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed

Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641482669

   > @Sn0rt It is very difficult to promote the change of the gateway in the production environment. Recently, there have been several failures in the infrastructure. Everyone is very careful when changing the infrastructure.
   
   In the production environment, can you check the log of the gateway? Is there any log of ERROR level?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] hellojukay commented on issue #9857: help request: docker pull huge image failed

Posted by "hellojukay (via GitHub)" <gi...@apache.org>.
hellojukay commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641905093

   > `2023-07-18_00-00-00__error.log:2023/07/17 09:30:38 [warn] 48#48: *75095135 an upstream response is buffered to a temporary file /usr/local/apisix/proxy_temp/0/83/0000017830 while reading upstream, client: 10.100.11.44, server: _, request: "GET /v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c HTTP/1.1", upstream: "http://10.100.25.63:8082/artifactory/api/docker/docker/v2/jadt/iii/blobs/sha256:fa92f16621a41cf669c82b37ff8ee85bf1541e2c9bc12d7943943e98a235c68c", host: "docker.xyz.com"`
   > 
   > ![image](https://user-images.githubusercontent.com/2706161/254556485-24161061-401f-41c7-bffc-ef839be34fcf.png)
   > 
   > pls take a look this link https://apisix.apache.org/docs/apisix/plugins/proxy-control/
   
   Should i disable `response buffer` , it not releated to `request buffer` ,  Am I understanding this correctly? As described in this article  https://segmentfault.com/a/1190000021704996


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed

Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641881106

   > [apisix.log](https://github.com/apache/apisix/files/12094189/apisix.log) this is the error log (merge by 2 pods.)
   
   if you can attach warn level log is better


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Sn0rt commented on issue #9857: help request: docker pull huge image failed

Posted by "Sn0rt (via GitHub)" <gi...@apache.org>.
Sn0rt commented on issue #9857:
URL: https://github.com/apache/apisix/issues/9857#issuecomment-1641367164

   hi [hellojukay](https://github.com/hellojukay), can you set the log level with debug? and take a look at the error.log file. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org