You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "wklken (via GitHub)" <gi...@apache.org> on 2023/08/28 08:23:03 UTC

[GitHub] [apisix] wklken opened a new issue, #10093: bug: dns resolution did not resume immediately after the dns server resume

wklken opened a new issue, #10093:
URL: https://github.com/apache/apisix/issues/10093

   ### Current Behavior
   
   In our production, when the dns server reload frequently.
   The requests will be all fail, while the dns resolution not ok `failed to parse domain: test.com, error: failed to query the DNS server: dns client error: 101 empty record received`;
   
   But when the dns server is stable, the dns resolution of apisix did not resume immediately, the requests got error `[lua] init.lua:486: handle_upstream(): failed to set upstream: no valid upstream node` for hours, until we do `apisix reload`, the dns turns ok.
   
   And from the log, one apisix instance with 4 processes, 3 processes got the dns failed, 1 processes got no error all the requests are ok.
   
   ---
   
   apisix/core/dns/client.lua
   
   ```lua
   function _M.resolve(self, domain, selector)
       local client = self.client
   
       local answers, err = client.resolve(domain)
       if not answers then
           return nil, "failed to query the DNS server: " .. err
       end
   ```
   
   how the `client.resolve` detect the dns server is ok? 
   
   
   
   
   
   ### Expected Behavior
   
   dns resolution resume immediately
   
   ### Error Logs
   
   [lua] upstream.lua:79: parse_domain_for_nodes(): dns resolver domain: test.com error: failed to query the DNS server: dns client error: 101 empty record received
   [lua] resolver.lua:80: parse_domain(): failed to parse domain: test.com, error: failed to query the DNS server: dns client error: 101 empty record received
   
   ### Steps to Reproduce
   
   For now, I cannot reproduce it, but it has occurred three times in our production.
   Any clues or advice for making dns server reload frequently?
   
   ### Environment
   
   - APISIX version (run `apisix version`): 3.2.1
   - Operating system (run `uname -a`):
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`):  nginx version: openresty/1.21.4.1
   - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
   - APISIX Dashboard version, if relevant:
   - Plugin runner version, for issues related to plugin runners:
   - LuaRocks version, for installation issues (run `luarocks --version`):
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205655825

   cache issue is unlikely because api_ctx is created newly for each new request. So for next request, new api_ctx will be created and the matched route should contain the correct routes configuration with the nodes field.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] jiangfucheng commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "jiangfucheng (via GitHub)" <gi...@apache.org>.
jiangfucheng commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1725524437

   @wklken Thanks for your detailed description. Based on your description, I think there are two issues that need to be answered.
   1. Why APISIX will print error log with `failed to set upstream: no valid upstream node` after parse domain failed.
   I think you already found the reason for this: `parse_domain_for_nodes()` function should return `nil, error` if dns parse failed.
   
   https://github.com/apache/apisix/blob/9b2031a61c8af9c010a923eea89c0dd029e0d971/apisix/utils/upstream.lua#L64-L86
   
   2. Why the `http_access_phase()` function print the matched route with a `nil` node field in your second request?
   https://github.com/apache/apisix/blob/9b2031a61c8af9c010a923eea89c0dd029e0d971/apisix/init.lua#L494-L504
   
   I think it was caused by `api_ctx.matched_route = route` this line, it changed the route config in memory. (Note that I think it did not just change the copy table but changed the origin route table).
   
   For the above two issues, I think we just fix the return value in `parse_domain_for_nodes` function enough.
   
   For other issues, I will continue to try to debug the code to give my opinion. 
   
   Another, I have a question, in the above description, you commented `handle_upstream  will never call parse_domain_for_nodes function` in the second request,  How do you get the conclusion of this, I find the `parse_domain_for_nodes` function is not record related logs to indicate the function has been executed. 
   
   (I have been quite busy with work recently, and I will help you troubleshoot this issue as soon as possible in my spare time)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1726197902

   @jiangfucheng `api_ctx.matched_route` is per request so even if it is set wrong, the subsequent request will set it again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205532352

   @Revolyssup Yes, with the patch, no 503 shows


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1707604699

   it's the plugin `proxy-rewrite` cause the error. if remove the plugin, no error!
   
   ```
       "plugins": {
         "proxy-rewrite": {
           "method": "GET",
           "uri": "/get",
           "use_real_request_uri_unsafe": false
         }
       }
   ```
   
   I will use apisix docker-compose to reproduce the problem.
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] jiangfucheng commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "jiangfucheng (via GitHub)" <gi...@apache.org>.
jiangfucheng commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1726773115

   > @wklken I think parse_domain_for_nodes is called in subsequent request but there are no nodes to iterate over, so you don't get the log.
   
   Agree with this opinion.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] jiangfucheng commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "jiangfucheng (via GitHub)" <gi...@apache.org>.
jiangfucheng commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1710270623

   @wklken  Hi, could you paste the APISIX configurations? It's hard to reproduce this bug if no detailed configurations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1724748805

   @jiangfucheng Finally, It been reproduced on the apisix docker-compose. It take a lot of time , add I add some scripts to help.
   
   here is the steps, can you please help to investigate this? (this is the problem which stuck our version for production)
   
   ------
   
   ## Reproduce steps:
   
   ### 1. use the docker-compose
   
   1. `git clone https://github.com/apache/apisix-docker.git`
   2. `vim  apisix_conf/master/config.yaml`, change the `router` and `nginx_config.processes`
   ```yaml
   apisix:
     node_listen: 9080              # APISIX listening port
     enable_ipv6: false
   
     router:
       http: radixtree_uri_with_parameter
   
   nginx_config:
     worker_processes: 4
   ```
   3. `vim docker-compose-master.yaml` change the `image` to 3.2.1-centos
   ```yaml
   services:
     apisix:
       image: "apache/apisix:3.2.1-centos"
   ```
   4. `docker-compose -p docker-apisix -f docker-compose-master.yaml up` start the apisix and etcd
   
   ### 2. enter the container, do some change to mock the dns failure
   
   > all in the container
   
   1. `docker exec -it dockerapisix_apisix_1 /bin/bash`
   2. check the `conf/config.yaml` is right, 4 processes, and the `apisix version` is 3.2.1
   3. `vi apisix/utils/upstream.lua`, add code to mock the dns failure, 50% will `101 empty records`
   
   ```lua
   -- to line 22
   
   local random = math.random
   
   -- add before core.resolver.parse_domain(host)
   
   local function parse_domain_for_nodes(nodes)
       local new_nodes = core.table.new(#nodes, 0)
       for _, node in ipairs(nodes) do
           local host = node.host
           if not ipmatcher.parse_ipv4(host) and
                   not ipmatcher.parse_ipv6(host) then
               if host == "httpbin.org" and ngx_now() <= 1695089345 then
                   if random(1,10) % 2 == 0 then
                           core.log.error("101 empty records")
                           return new_nodes
                   end
               end
               local ip, err = core.resolver.parse_domain(host)
   ```
   4. add a script `create.sh` to register the route and service, **check the API_KEY is right**
   ```bash
   #!/bin/bash
   
   API_KEY="edd1c9f034335f136f87ad84b625c8f1"
   ROUTE_ID="dns_route"
   SERVVICE_ID="dns_service"
   
   curl http://127.0.0.1:9180/apisix/admin/services/${SERVVICE_ID} -H "X-API-KEY: ${API_KEY}" -X PUT -d '
   {
     "upstream": {
       "nodes": [
         {
           "host": "httpbin.org",
           "port": 80,
           "weight": 100,
           "priority": 1
         }
       ],
       "type": "roundrobin",
       "scheme": "http",
       "pass_host": "node"
     }
   }'
   
   curl http://127.0.0.1:9180/apisix/admin/routes/${ROUTE_ID} -H "X-API-KEY: ${API_KEY}" -X PUT -d '
   {
     "uri": "/api/test/prod/dns22",
     "methods": [
       "GET"
     ],
     "plugins": {
       "proxy-rewrite": {
         "method": "GET",
         "uri": "/get"
       }
     },
     "upstream": {
       "nodes": [
         {
           "host": "httpbin.org",
           "port": 80,
           "weight": 100,
           "priority": 1
         }
       ],
       "type": "roundrobin",
       "scheme": "http",
       "pass_host": "node"
     },
     "service_id": "dns_service",
     "status": 1
   }'
   ```
   5. add script `update_lua.sh` to change the timestamp fast
   ```bash
   #!/bin/bash
   
   now=$(date "+%s")
   echo "now is: ${now}"
   
   A=$(date -d "+30 seconds" "+%s")
   echo "will change the condition to <= ${A}"
   sed -i -r "s/<= ([0-9]+) then/<= ${A} then/g"  apisix/utils/upstream.lua
   echo $?
   echo "change done"
   
   apisix reload
   ```
   6. do the register
   ```bash
   bash -x create.sh
   ```
   7. check the url is ok **not in the container**
   ```
   curl -vv http://0.0.0.0:9080/api/test/prod/dns22
   ```
   
   ## 3. not in the container, add check script
   
   > not in the container
   
   1. install wrk
   2. add script `start_and_check.sh` for benchmark then check 503
   ```bash
   #!/bin/bash
   
   date
   
   url="http://0.0.0.0:9080/api/test/prod/dns22"
   
   echo "start bench wrk"
   wrk -c2 -t2 -d35s ${url}
   
   date
   
   echo "sleep 5 s"
   sleep 5
   
   echo "check the status code 10 times"
   
   for ((i=1; i<=10; i++))
   do
       status_code=$(curl --write-out %{http_code} --silent --output /dev/null $url)
       echo "status=$status_code"
       if [ "${status_code}" -eq "503" ]
       then
           echo "503 show"
           exit
       fi
   done
   ```
   
   ## 4. reproduce
   
   > open two window to run step 1/2 at the same time
   
   1. in container, run 
   ```
   $ ./update_lua.sh 
   now is: 1695089267
   will change the condition to <= 1695089297
   0
   change done
   /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua reload
   
   WARNING: using fixed Admin API token has security risk.
   Please modify "admin_key" in conf/config.yaml .
   
   
   2023/09/19 02:07:47 [notice] 157#157: signal process started
   ```
   2. out the container, run
   ```
   $ bash start_and_check.sh
   
   Tue Sep 19 10:08:36 CST 2023
   start bench wrk
   Running 35s test @ http://0.0.0.0:9080/api/test/prod/dns22
     2 threads and 2 connections
     Thread Stats   Avg      Stdev     Max   +/- Stdev
       Latency    15.22ms   80.61ms 823.04ms   95.99%
       Req/Sec     6.91k   807.96     7.77k    95.97%
     461582 requests in 35.00s, 202.93MB read
     Non-2xx or 3xx responses: 461575
   Requests/sec:  13187.64
   Transfer/sec:      5.80MB
   Tue Sep 19 10:09:11 CST 2023
   sleep 5 s
   check the status code 10 times
   status=503
   503 show
   ```
   3. if `503 show`, means it been reproduced! you can `curl -vv http://0.0.0.0:9080/api/test/prod/dns22` for few times to check
   ```
   curl -vv http://0.0.0.0:9080/api/test/prod/dns22
   * About to connect() to 0.0.0.0 port 9080 (#0)
   *   Trying 0.0.0.0...
   * Connected to 0.0.0.0 (0.0.0.0) port 9080 (#0)
   > GET /api/test/prod/dns22 HTTP/1.1
   > User-Agent: curl/7.29.0
   > Host: 0.0.0.0:9080
   > Accept: */*
   > 
   < HTTP/1.1 503 Service Temporarily Unavailable
   < Date: Tue, 19 Sep 2023 02:21:58 GMT
   < Content-Type: text/html; charset=utf-8
   < Content-Length: 269
   < Connection: keep-alive
   < Server: APISIX/3.2.1
   < 
   <html>
   <head><title>503 Service Temporarily Unavailable</title></head>
   <body>
   <center><h1>503 Service Temporarily Unavailable</h1></center>
   <hr><center>openresty</center>
   <p><em>Powered by <a href="https://apisix.apache.org/">APISIX</a>.</em></p></body>
   </html>
   * Connection #0 to host 0.0.0.0 left intact
   ```
   5. if all `200`, do step 1/2 again(it took 50 seconds each time, and it not 100% show 503, but if you try about 10 times, high possibility showed).
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205859094

   ![Screenshot from 2024-07-03 16-59-17](https://github.com/apache/apisix/assets/43276904/1f837916-b325-46c5-a15d-df9499f2cb3e)
   Tried multiple times, still getting no 503s 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "sheharyaar (via GitHub)" <gi...@apache.org>.
sheharyaar commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1888689921

   Hi @wklken , can you please recheck if this is fixed or not. I think this may have been related to #10722 .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1905887931

   @sheharyaar Unfortunately, the bug is not fixed by #10722 
   
   Follow the reproduce steps here: https://github.com/apache/apisix/issues/10093#issuecomment-1724748805
   
   1. reproduce, 503 show
   2. patch the changes, apisix reload
   3. reproduce, 503 show
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1726375021

   @wklken I think parse_domain_for_nodes is called in subsequent request but there are no nodes to iterate over, so you don't get the log.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1727132364

   @wklken Just one last request, can you add a log to log the route in the for loop here https://github.com/apache/apisix/blob/a98f1ca7a4b368a5270c5cae2a2a526f21ba409a/apisix/http/route.lua#L40 ? and show the output. I am trying to figure out at what point exactly did the route change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1724947998

   I add some log,  the image is more clear
   
   <img width="1867" alt="image" src="https://github.com/apache/apisix/assets/2002216/2b55197b-9b16-4f67-b105-9fdf59784e88">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1727199314

   > @wklken In the first request, did you get a log similar to `route items: ` or `insert uri route: `???
   
   My log level is `info` and there no `route items:` or `insert uri route:` where the first `empty nodes` present. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1727205200

   @wklken Before empty nodes came, did you find this? What was the last time this log came?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205464786

   > [#10093 (comment)](https://github.com/apache/apisix/issues/10093#issue-1869257753)
   > 
   > ```
   > The requests will be all fail, while the dns resolution not ok failed to parse domain: test.com, error: failed to query the DNS server: dns client error: 101 empty record received;
   > 
   > But when the dns server is stable, the dns resolution of apisix did not resume immediately, the requests got error [lua] init.lua:486: handle_upstream(): failed to set upstream: no valid upstream node for hours, until we do apisix reload, the dns turns ok.
   > ```
   > 
   > It’s always showing a 503 error even if the DNS is okay. I think there may be some cache that makes APISIX unaware of the DNS resume.
   
   @wklken I meant to ask the behavior in your patch. After your patch, you do not get the 503 after dns server resumes. Right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1711276625

   @jiangfucheng I still trying to reproduce it in apisix docker-compose, not success yet. I will update the issue when success. (it only can be reproduced on our image build from `apache/apisix:3.2.1-centos`)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1727143666

   @wklken In the first request, did you get a log similar to `route items: `???


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1701079886

   two upstream:
   1. route.upstream
   2. route.service_id -> service.upstream
   
   the first upstream did not resume immediately
   
   https://github.com/apache/apisix/blob/77704832ec91117f5ca7171811ae5f0d3f1494fe/apisix/init.lua#L422-L445
   
   and it seems the `parse_domain_in_route` did not know the dns server is resumed
   
   https://github.com/apache/apisix/blob/77704832ec91117f5ca7171811ae5f0d3f1494fe/apisix/init.lua#L210-L239


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2204851831

   @Revolyssup did you follow those steps here? https://github.com/apache/apisix/issues/10093#issuecomment-1724748805
   
   and the apisix version should be 3.2.1;
   
   ----
   
   we just patch it https://github.com/TencentBlueKing/blueking-apigateway-apisix/blob/master/src/build/patches/002_upstream_parse_domain_for_nodes.patch and it's on our production for about 10 months, and it works.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] jiangfucheng commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "jiangfucheng (via GitHub)" <gi...@apache.org>.
jiangfucheng commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1701243254

   Hi, thank you for your report. Could you give us a more detailed description of your configuration?
   
   1. What is the local DNS server you are running locally?  And we need the related configuration.
   2. Could you paste your related configuration of APISIX?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] jiangfucheng commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "jiangfucheng (via GitHub)" <gi...@apache.org>.
jiangfucheng commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1726770376

   > @jiangfucheng `api_ctx.matched_route` is per request so even if it is set wrong, the subsequent request will set it again. In the next request, you can see that api_ctx.matched_route has empty nodes in upstream after it is populated by match() function. The whole issue boils down to, why the matched_route log has no nodes? This is calculated for each request from the etcd. I haven't been able to find the exact flow that can explain where exactly did this nodes={} from previous upstream resulted in api_ctx.matched_route's nodes when api_ctx is freshly calculated per request. And where exactly is the caching happening at that point while matching route.
   > 
   > The only caching culprit I see currently is this
   > 
   > https://github.com/apache/apisix/blob/57a7dcdac63a0bda55ddd41fc56a10c6144bfd51/apisix/http/router/radixtree_uri_with_parameter.lua#L38
   > 
   > But modification in _M.user_routes is only done directly by etcd.
   
   
   
   You are right, it's my mistake, `api_ctx` will be created when a new request comes, it does not reuse it. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1726344183

   @wklken Just to confirm, it only happens when the route has service id, right?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205396137

   @wklken Just want to confirm the behaviour after this patch.
   So before the patch when empty nodes were returned, you were intermittently getting 503 error with the requests.
   But now when you errror out early, you still get 500 error for "no valid ip found". But the subsequent requests when the dns comes back up, no error is found. right?
   
   Basically you have covered the case that nodes will never be set as empty.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205452606

   https://github.com/apache/apisix/issues/10093#issue-1869257753
   
   ```
   The requests will be all fail, while the dns resolution not ok failed to parse domain: test.com, error: failed to query the DNS server: dns client error: 101 empty record received;
   
   But when the dns server is stable, the dns resolution of apisix did not resume immediately, the requests got error [lua] init.lua:486: handle_upstream(): failed to set upstream: no valid upstream node for hours, until we do apisix reload, the dns turns ok.
   ```
   
   It’s always showing a 503 error even if the DNS is okay.
   I think there may be some cache that makes APISIX unaware of the DNS resume.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1888717113

   @sheharyaar ok, I will recheck it later


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "sheharyaar (via GitHub)" <gi...@apache.org>.
sheharyaar commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1905902572

   Ok, I will have a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205602580

   I haven't been able to reproduce but by the error and how your patch fixes the error and the logs above that you showed, here is the conclusion so far:
   
   Somehow the nodes in the original saved configuration are set to empty. Since this goes away after a reload means the etcd still has correct value.  Since I cant reproduce, I am going through the code to find the root cause currently.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup closed issue #10093: bug: dns resolution did not resume immediately after the dns server resume
URL: https://github.com/apache/apisix/issues/10093


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1754298110

   Closed by https://github.com/TencentBlueKing/blueking-apigateway-apisix/pull/52 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup closed issue #10093: bug: dns resolution did not resume immediately after the dns server resume
URL: https://github.com/apache/apisix/issues/10093


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1754326938

   @wklken Yep I understand. The patch still doesn't fix the original issue, its kind of a hack


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1730902811

   cc: @kingluo 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1702146347

   I have reproduce @jiangfucheng 
   
   just mock dns fail in lua
   
   ## apisix config:
   
   1. start apisix with 4 processes
   2. change then nginx error log level to `info`
   
   ## register route:
   
   1. add a route `/get` with one upstream, one node like `httpbin.org`; the parsed data in log is `"nodes":[{"host":"54.175.87.239","priority":1,"domain":"httpbin.org","port":80,"weight":100,"upstream_host":"httpbin.org"}]`
   2. change `apisix/utils/upstream.lua` to mock dns random error, note the `1693542089` is a `now() + 60`
   
   ```lua
   local function parse_domain_for_nodes(nodes) 
       local new_nodes = core.table.new(#nodes, 0)
       for _, node in ipairs(nodes) do
           local host = node.host
           if not ipmatcher.parse_ipv4(host) and
                   not ipmatcher.parse_ipv6(host) then
   
               if host == "httpbin.org" and ngx_now() <= 1693542089 then
                   if random(1,10) % 2 == 0 then
                           core.log.error("101 empty records")
                           return new_nodes
                   end
                   -- return nil, "failed to query the DNS server: timeout11"
               end
               local ip, err = core.resolver.parse_domain(host)
               if ip then
                   local new_node = core.table.clone(node)
                   new_node.host = ip
                   new_node.domain = host
                   core.table.insert(new_nodes, new_node)
               end
   ```
   
   3. `apisix reload`
   
   ## do requests
   
   1. `wrk  -c2 -t2 -d60s http://{}/get`
   2. and after the wrk finished, do `curl http://{}/get` for about 10 times, got 200 and 503
   
   ## the log
   
   when 503, there got no `dns` logs before `no valid upstream node`
   
   
   ```
   2023/09/01 04:24:20 [info] 140701#140701: *15533385 [lua] radixtree_uri_with_parameter.lua:53: match(): route match mode: radixtree_uri_with_parameter
   2023/09/01 04:24:20 [info] 140701#140701: *15533385 [lua] init.lua:572: http_access_phase(): matched route: {"key":"/bk-gateway-apisix/routes/test.prod.2423","modifiedIndex":532500,"clean_handlers":"table: 0x7f7d645c64c8","has_domain":true,"value":{"create_time":1693309224,"methods":["GET"],"service_id":"test.prod.stage-12","update_time":1693539224,"timeout":{"send":150,"read":150,"connect":150},"labels":{"gateway.bk.tencent.com/gateway":"test","gateway.bk.tencent.com/stage":"prod"},"uris":["/api/test/prod/dns","/api/test/prod/dns/"],"status":1,"priority":0,"name":"test-prod-dns","id":"test.prod.2423","plugins":{"bk-proxy-rewrite":{"match_subpath":false,"subpath_param_name":":ext","method":"GET","use_real_request_uri_unsafe":false,"uri":"/get"},"bk-resource-context":{"bk_resource_auth":{"resource_perm_required":false,"skip_user_verification":false,"verified_app_required":false,"verified_user_required":false},"bk_resource_id":2423,"bk_resource_name":"dns","bk_resource_auth_obj":{"r
 esource_perm_required":false,"skip_user_verification":false,"verified_app_required":false,"verified_user_required":false}}},"upstream":{"nodes":{},"hash_on":"vars","scheme":"http","parent":{"key":"/bk-gateway-apisix/routes/test.prod.2423","modifiedIndex":532500,"clean_handlers":{},"has_domain":true,"value":"table: 0x7f7d9706c708","update_count":0,"orig_modifiedIndex":532500,"createdIndex":532332},"type":"roundrobin","pass_host":"node","timeout":{"send":150,"read":150,"connect":150}}},"update_count":0,"orig_modifiedIndex":532500,"createdIndex":532332
   2023/09/01 04:24:20 [info] 140701#140701: *15533385 [lua] plugin.lua:630: merge_service_route():   route conf: {"key":"/bk-gateway-apisix/routes/test.prod.2423","modifiedIndex":532500,"clean_handlers":"table: 0x7f7d645c64c8","has_domain":true,"value":{"create_time":1693309224,"methods":["GET"],"service_id":"test.prod.stage-12","update_time":1693539224,"timeout":{"send":150,"read":150,"connect":150},"labels":{"gateway.bk.tencent.com/gateway":"test","gateway.bk.tencent.com/stage":"prod"},"uris":["/api/test/prod/dns","/api/test/prod/dns/"],"status":1,"priority":0,"name":"test-prod-dns","id":"test.prod.2423","plugins":{"bk-proxy-rewrite":{"match_subpath":false,"subpath_param_name":":ext","method":"GET","use_real_request_uri_unsafe":false,"uri":"/get"},"bk-resource-context":{"bk_resource_auth":{"resource_perm_required":false,"skip_user_verification":false,"verified_app_required":false,"verified_user_required":false},"bk_resource_id":2423,"bk_resource_name":"dns","bk_resource_auth_obj":
 {"resource_perm_required":false,"skip_user_verification":false,"verified_app_required":false,"verified_user_required":false}}},"upstream":{"nodes":{},"hash_on":"vars","scheme":"http","parent":{"key":"/bk-gateway-apisix/routes/test.prod.2423","modifiedIndex":532500,"clean_handlers":{},"has_domain":true,"value":"table: 0x7f7d9706c708","update_count":0,"orig_modifiedIndex":532500,"createdIndex":532332},"type":"roundrobin","pass_host":"node","timeout":{"send":150,"read":150,"connect":150}}},"update_count":0,"orig_modifiedIndex":532500,"createdIndex":532332}
   2023/09/01 04:24:21 [error] 140701#140701: *15534047 [lua] init.lua:486: handle_upstream(): failed to set upstream: no valid upstream node
   ```
   
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1724758087

   More clues for detecting:
   1. should be `route.upstream`, and the `route` should related to a `service` with `upstream` will cause the problem
       - route without upstream - ok
       - route with upstream, without service_id - ok
   2. 4 workers, only part of them will return 503
   3. if `apisix reload`, will all 200
   4. the `503` only 1 log `2023/09/19 02:34:52 [error] 854#854: *112694 [lua] init.lua:486: handle_upstream(): failed to set upstream: no valid upstream node, client: 172.31.0.1, server: _, request: "GET /api/test/prod/dns22 HTTP/1.1", host: "0.0.0.0:9080"`; it did not call the `parse_domain`, maybe some cache? 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1754323614

   @Revolyssup the issue should not be closed before the official repo fix the bug? and we still don't know the reason.
   
   our own repo need to fix the production so we patched it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] Revolyssup commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1726114490

   @jiangfucheng , in the log you can see that @wklken has added I think some error logs from his side `101 empty record` inside `parse_domain_for_nodes` function. 
   
   @wklken Can you show the snippet of `parse_domain_for_nodes` function where this log is?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1726766791

   > @wklken Just to confirm, it only happens when the route has service id, right?
   
   Yes, route has upstream and has service_id (the service have upstream too)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1702294146

   `route.value.upstream.nodes` is empty, so the `new_nodes` is empty(no dns resolve)
   
   <img width="862" alt="image" src="https://github.com/apache/apisix/assets/2002216/b883927c-f650-4e1d-847b-3019083c6293">
   
   -----
   
   so the `route.value.upsteram.nodes` may be set empty somewhere
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [apisix] wklken commented on issue #10093: bug: dns resolution did not resume immediately after the dns server resume

Posted by "wklken (via GitHub)" <gi...@apache.org>.
wklken commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-1738381865

   https://github.com/apache/apisix/issues/10093#issuecomment-1725524437
   
   > For the above two issues, I think we just fix the return value in `parse_domain_for_nodes` function enough.
   
   @jiangfucheng we need your help to confirm if that fix would be right.
   
   According to your answer above, can we just add a check before `return new_nodes` inf `local function parse_domain_for_nodes(nodes)`? 
   
   ```
   local function parse_domain_for_nodes(nodes)
       .....
   
       if #new_nodes == 0 then
         return nil, "no valid nodes after dns resolver domain: " .. host
       end
   
       return new_nodes
   
   ```
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2203125079

   @wklken I have followed your exact reproduction steps multiple times but I never got the error. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205869452

   @wklken Also one clarification in the reproduction steps.
   
   You ask to run  wrk just after running update_lua,sh. If you do that then 503 are expected because for the first 30 seconds the custom added code will sometimes return 503, the code that is added for reproduction. But after 30 seconds there can be no issue as that code doesn't run. 
   ![Screenshot from 2024-07-03 17-03-37](https://github.com/apache/apisix/assets/43276904/cdc2c9a8-03c6-4bae-96e5-d82dbd3be01c)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2208426212

   Since after trying multiple times, this cannot be reproduced. I am closing the issue. If anyone else is able to reproduce this bug again then please reopen with the exact steps you folllowed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: dns resolution did not resume immediately after the dns server resume [apisix]

Posted by "Revolyssup (via GitHub)" <gi...@apache.org>.
Revolyssup commented on issue #10093:
URL: https://github.com/apache/apisix/issues/10093#issuecomment-2205175830

   > @Revolyssup did you follow those steps here? https://github.com/apache/apisix/issues/10093#issuecomment-1724748805
   > 
   > 
   > 
   > and the apisix version should be 3.2.1;
   > 
   > 
   > 
   > ----
   > 
   > 
   > 
   > we just patch it https://github.com/TencentBlueKing/blueking-apigateway-apisix/blob/master/src/build/patches/002_upstream_parse_domain_for_nodes.patch and it's on our production for about 10 months, and it works.
   > 
   > 
   > 
   > 
   
   Yes, I had followed the steps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org