You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "jinjianming (via GitHub)" <gi...@apache.org> on 2023/03/10 04:29:11 UTC

[GitHub] [apisix] jinjianming opened a new issue, #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

jinjianming opened a new issue, #9049:
URL: https://github.com/apache/apisix/issues/9049

   ### Description
   
   At present, I have tried to deploy in docker-compose and helm chart mode, but there are problems with routing failure and upstream address failure to update;
   
   Next, I will describe how the docker-compose deployment method replicates the problem, as well as the entire attachment information of my docker-compose.
   
   ```
   1.git clone https://github.com/apache/apisix-docker.git
   
   2.Modify the configuration and change it to Decoupled mode. See the attachment (including certificate file)
   
   3.Authorization log related files
   
   `chmod 777 apisix_cp_log/ apisix_dp_log/  certs/ -R`
   
   4.Start Service 
   `docker-compose -p docker-api up -d`
   
   5.Create route
   ---
   curl -X PUT http://127.0.0.1:9180/apisix/admin/routes/1     -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -i -d '
   {
       "methods": ["GET"],
       "uri": "/web1",
       "upstream": {
           "type": "roundrobin",
           "nodes": {
               "web1:80": 1
           }
       }
   }'
   ---
   6.Request DP service, return 404
   
   [root@master01 example]# curl http://127.0.0.1:9080/web1
   {"error_msg":"404 Route Not Found"}
   7.Confirm that the route write succeeded
   ![image](https://user-images.githubusercontent.com/57084209/224220993-e8727c19-dc5c-49a7-bf54-104e26cfd148.png)
   8.Restart DP service
   `docker restart docker-apisix_apisix-dp_1 `
   9.Request again to return to normal.
   [root@master01 example]# curl http://127.0.0.1:9080/web1
   hello web1
   ```
   ---
   So far, if the DP service is not restarted, the routing cannot take effect
   
   I don't know whether there is a problem with my configuration. I have studied this problem for a long time and still haven't made any progress
   [example.zip](https://github.com/apache/apisix/files/10938507/example.zip)
   ;
   
   
   ### Environment
   
   - APISIX version (run `apisix version`):  3.2.0
   - Operating system (run `uname -a`): `Linux master01 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux`
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): nginx version: openresty/1.21.4.1
   - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): `{"hostname":"162d98767e64","id":"8cd6dc41-d64f-4d5d-ad0f-46a4ec423bf0","boot_time":1678421364,"etcd_version":"unknown","version":"3.2.0"}`
   - APISIX Dashboard version, if relevant:
   - Plugin runner version, for issues related to plugin runners:
   - LuaRocks version, for installation issues (run `luarocks --version`):
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] jinjianming commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "jinjianming (via GitHub)" <gi...@apache.org>.
jinjianming commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1463761896

   Can someone help me look at this problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] kingluo closed issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo closed issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information
URL: https://github.com/apache/apisix/issues/9049


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] jinjianming commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "jinjianming (via GitHub)" <gi...@apache.org>.
jinjianming commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1482164234

   > @jinjianming你找到解决办法了吗？我也面临这个问题。
   
   No, I now deploy APISIX using a traditional architecture;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] jinjianming commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "jinjianming (via GitHub)" <gi...@apache.org>.
jinjianming commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1485058878

   > sync_data
   
   Yes, don't know why `res: null`. Under normal circumstances, the upstream change log should contain the following contents:
   ```
   2023/03/27 12:29:38 [info] 55#55: *858 [lua] config_etcd.lua:389: sync_data(): waitdir key: /apisix/upstreams prev_index: 39693, context: ngx.timer
   2023/03/27 12:29:38 [info] 55#55: *858 [lua] config_etcd.lua:390: sync_data(): res: {"body":{"node":[{"value":{"id":"16690fc3","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","pass_host":"pass","name":"default_js-design-nginx_80","nodes":{},"update_time":1679920178,"create_time":1679543231,"labels":{"managed-by":"apisix-ingress-controller"},"scheme":"http","type":"roundrobin","hash_on":"vars"},"modifiedIndex":39693,"key":"/apisix/upstreams/16690fc3","createdIndex":10837}]},"headers":{"X-Etcd-Index":"39693"}}, err: nil, context: ngx.timer
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] xywang0471 commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "xywang0471 (via GitHub)" <gi...@apache.org>.
xywang0471 commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1516103383

   @kingluo 我这里使用的是2.15版本，我的问题是upstream中的node列表更新后，apisix使用的还是旧的node列表。我读了源码后发现这里可能是个问题。图中 server_picker 使用了 lrucache 缓存，key 的版本号是route的modifiedIndex，这就会导致在route没有更新，但upstream的node列表更新了不能及时同步。
   ![image](https://user-images.githubusercontent.com/32950283/233338761-dcfff633-0543-4f37-8b7b-7597fa7221fb.png)
   ![image](https://user-images.githubusercontent.com/32950283/233341244-a0f27b83-fac6-4849-8948-9c43b4d6e52d.png)
   ![image](https://user-images.githubusercontent.com/32950283/233341394-2f176c7c-5102-4ded-b973-5fd5849ec423.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] kingluo commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1508231426

   Yes, exactly. This directive is configured on CP only.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] kingluo commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1704311140

   The conf server is removed, so the etcd communication rolls back to what 2.x versions behave like, i.e. no matter CP or DP, connects to etcd directly. Closed by https://github.com/apache/apisix/pull/10012


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] Baoyuantop commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1484835227

   Found this information in apisix_dp_log:
   
   ```
   2023/03/27 09:41:11 [info] 74#74: *59 [lua] config_etcd.lua:389: sync_data(): waitdir key: /apisix/routes prev_index: 240, context: ngx.timer
   2023/03/27 09:41:11 [info] 74#74: *59 [lua] config_etcd.lua:390: sync_data(): res: null, err: timeout, context: ngx.timer
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] jinjianming commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "jinjianming (via GitHub)" <gi...@apache.org>.
jinjianming commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1485058726

   > sync_data
   
   Yes, don't know why `res: null`. Under normal circumstances, the upstream change log should contain the following contents:
   ```
   2023/03/27 12:29:38 [info] 55#55: *858 [lua] config_etcd.lua:389: sync_data(): waitdir key: /apisix/upstreams prev_index: 39693, context: ngx.timer
   2023/03/27 12:29:38 [info] 55#55: *858 [lua] config_etcd.lua:390: sync_data(): res: {"body":{"node":[{"value":{"id":"16690fc3","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","pass_host":"pass","name":"default_js-design-nginx_80","nodes":{},"update_time":1679920178,"create_time":1679543231,"labels":{"managed-by":"apisix-ingress-controller"},"scheme":"http","type":"roundrobin","hash_on":"vars"},"modifiedIndex":39693,"key":"/apisix/upstreams/16690fc3","createdIndex":10837}]},"headers":{"X-Etcd-Index":"39693"}}, err: nil, context: ngx.timer
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] jinjianming commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "jinjianming (via GitHub)" <gi...@apache.org>.
jinjianming commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1508132911

   > > 这种临时方法有什么隐患吗？我想把它部署到我们的生产环境；
   > 
   > @jinjianming没有危险。只是对性能有一点影响。也就是说，每个响应都不会被缓冲并从上游服务器发送到下游客户端。
   
   Thank you. @kingluo 
   
   `proxy_ buffering off;` Add this configuration CP or DP there;
   
   If it is only configured in CP, it should not affect the performance of DP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] kingluo commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1534160429

   @xywang0471 The issue you mentioned seems another issue irrelevant to this one. So could you raise a separate issue report, and I will close this one?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] jinjianming commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "jinjianming (via GitHub)" <gi...@apache.org>.
jinjianming commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1507994010

   > @jinjianming In decoupled deployment mode, CP only provides https, for access by DP. But with `proxy_buffering on;` (by default), the SSL response (chunked watch result blocks) will be buffered and finally timeout.
   > 
   > We should fix it later. But the temporary solution is:
   > 
   > `config.yaml`
   > 
   > ```yaml
   > nginx_config:
   >   http_configuration_snippet: |
   >     proxy_buffering off;
   > ```
   > 
   > Similar to this issue, which proxies the etcd via APISIX: [#9263 (comment)](https://github.com/apache/apisix/issues/9263#issuecomment-1506475658)
   
   Is there any hidden danger in this temporary method? I want to deploy it to our production environment;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] kingluo commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1508009832

   > Is there any hidden danger in this temporary method? I want to deploy it to our production environment;
   
   @jinjianming No danger. Just has a little performance impact. That is, each response is not buffered and sent through to the downstream client.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] kingluo commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "kingluo (via GitHub)" <gi...@apache.org>.
kingluo commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1506487973

   @jinjianming 
   In decoupled deployment mode, CP only provides https, for access by DP. But with `proxy_buffering on;` (by default), the SSL response (chunked watch result blocks) will be buffered and finally timeout.
   
   We should fix it later. But the temporary solution is:
   
   `config.yaml`
   
   ```yaml
   nginx_config:
     http_configuration_snippet: |
       proxy_buffering off;
   ```
   
   Similar to this issue, which proxies the etcd via APISIX:
   https://github.com/apache/apisix/issues/9263#issuecomment-1506475658


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [apisix] PhanPirang commented on issue #9049: help request: Decoupled deployment mode DP module cannot get the latest routing and upstream information

Posted by "PhanPirang (via GitHub)" <gi...@apache.org>.
PhanPirang commented on issue #9049:
URL: https://github.com/apache/apisix/issues/9049#issuecomment-1480877769

   @jinjianming Have you found the solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org