You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "indrekj (via GitHub)" <gi...@apache.org> on 2025/02/25 08:26:50 UTC

[I] bug: limit-count rejections are not captured in the metrics [apisix]

indrekj opened a new issue, #11995:
URL: https://github.com/apache/apisix/issues/11995

   ### Current Behavior
   
   When the `limit-count` plugin rejects a request, there will be a log message, e.g.
   ```
   2025/02/25 08:16:10 [warn] 49#49: *42708330 [lua] plugin.lua:1153: run_plugin(): limit-count exits with http status code 429, client: 34.215.116.35, server: _, request: "POST /some-address HTTP/1.1", host: "[redacted]", referrer: "[redacted]"
   ```
   
   However, these "429" response status codes aren't in the metrics. I've tried it with both the [datadog](https://apisix.apache.org/docs/apisix/plugins/datadog/) plugin and the [prometheus](https://apisix.apache.org/docs/apisix/plugins/prometheus/) plugin.
   
   ### Expected Behavior
   
   _No response_
   
   ### Error Logs
   
   _No response_
   
   ### Steps to Reproduce
   
   1. Enable [limit-count](https://apisix.apache.org/docs/apisix/plugins/limit-count/) plugin
   2. Set up a route with strict rate limiting (set rejected_code to 429)
   3. Enable prometheus plugin
   4. Send a bunch of requests to the route so that it gets limited
   5. Query the response metrics (/apisix/prometheus/metrics). Expected: 429 responses, actual: no 429 responses
   
   ### Environment
   
   - Installed through the `Apache APISIX Helm Chart`
   - APISIX version (run `apisix version`): 3.7.0
   - Operating system (run `uname -a`): Linux apisix-76f84f94f7-blqrb 5.10.228-219.884.amzn2.x86_64 #1 SMP Wed Oct 23 17:17:00 UTC 2024 x86_64 GNU/Linux
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
     ```
   nginx version: openresty/1.21.4.2
   built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
   built with OpenSSL 1.1.1s  1 Nov 2022 (running with OpenSSL 1.1.1w  11 Sep 2023)
   TLS SNI support enabled
   configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_RUNTIME_VER=1.0.1 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.2 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.25 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.34 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-
 module=../ngx_stream_lua-0.0.13 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/stream --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/meta --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../wasm-nginx-module-0.6.5 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.gLDkH7DPEH/openresty-1.21.4.2/../grpc-
 client-nginx-module-v0.4.4 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: limit-count rejections are not captured in the metrics [apisix]

Posted by "indrekj (via GitHub)" <gi...@apache.org>.
indrekj commented on issue #11995:
URL: https://github.com/apache/apisix/issues/11995#issuecomment-2682248847

   This is not a bug actually, finally figured it out. We were overriding the error page handler:
   
   ```
         httpSrv: |
           access_log /dev/stdout main buffer=16384 flush=3 if=$loggable;
   
           location @error {
             default_type 'text/html';
             echo '<html><head><title>$status $status_text</title></head><body><center><h1>$status $status_text</h1></center></html>';
           }
           error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 421 422 423 424 425 426 428 429 431 451 500 501 502 503 504 505 506 507 508 510 511 @error;
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: limit-count rejections are not captured in the metrics [apisix]

Posted by "indrekj (via GitHub)" <gi...@apache.org>.
indrekj commented on issue #11995:
URL: https://github.com/apache/apisix/issues/11995#issuecomment-2685160517

   I solved it by writing a custom plugin:
   
   ```
     - apiVersion: v1
       kind: ConfigMap
       metadata:
         name: apisix-custom-plugins
       data:
         glia-errors.lua: |
           local core = require("apisix.core")
           local schema = {
             type = "object",
             properties = {},
             required = {},
           }
           local _M = {
             version = 0.1,
             priority = 10,
             name = "glia-errors",
             schema = schema,
           }
   
           function _M.check_schema(conf)
             return core.schema.check(schema, conf)
           end
   
           local function is_match()
             return ngx.status == 429
           end
   
           function _M.header_filter(conf, ctx)
             if is_match() then
               core.response.clear_header_as_body_modified()
             end
           end
   
           function _M.body_filter(conf, ctx)
             if is_match() then
               local body = core.response.hold_body_chunk(ctx)
               if not body then
                 return
               end
   
               ngx.arg[1] = "<html><head><title>" .. ngx.status .. " " .. ngx.var.status_text .. "</title></head><body><center><h1>" .. ngx.status .. " " .. ngx.var.status_text ..  "</h1></center></html>\n"
             end
           end
   
           return _M
   ```
   
   Note that this is just the preliminary code. It doesn't distinguish between apisix errors and upstream errors. But for 429 it is fine for us. Also it doesn't handle other >= 400 errors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] bug: limit-count rejections are not captured in the metrics [apisix]

Posted by "indrekj (via GitHub)" <gi...@apache.org>.
indrekj closed issue #11995: bug: limit-count rejections are not captured in the metrics
URL: https://github.com/apache/apisix/issues/11995


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org