hebbaa opened a new issue, #12769: URL: https://github.com/apache/apisix/issues/12769 ### Current Behavior Seeing below error in our production apisix data-plane logs 2025/11/23 10:46:52 [error] 54#54: *643711487 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer 2025/11/23 10:46:52 [error] 116#116: *643711540 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/events/compat/init.lua:47: attempt to index upvalue 'ev' (a nil value) 2025/11/23 10:46:52 [error] 52#52: *643711538 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer ### Expected Behavior These error logs should not be there. ### Error Logs 2025/11/23 10:46:52 [error] 54#54: *643711487 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer 2025/11/23 10:46:52 [error] 116#116: *643711540 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/events/compat/init.lua:47: attempt to index upvalue 'ev' (a nil value) 2025/11/23 10:46:52 [error] 52#52: *643711538 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer ### Steps to Reproduce Dont know. Happening in production deployment. ### Environment - APISIX version (run apisix version):3.14.1 - Operating system (run uname -a):Linux data-plane-7c4b955754-dp552 5.10.230-223.885.amzn2.x86_64 https://github.com/apache/apisix/pull/1 SMP Tue Dec 3 14:36:00 UTC 2024 x86_64 GNU/Linux - OpenResty / Nginx version (run openresty -V or nginx -V):nginx version: openresty/1.27.1.2 (x86_64-pc-linux-gnu) built by gcc 15.2.0 (Wolfi 15.2.0-r3) built with OpenSSL 3.6.0 1 Oct 2025 TLS SNI support enabled -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
Baoyuantop commented on issue #12769: URL: https://github.com/apache/apisix/issues/12769#issuecomment-3568764897 Hi @hebbaa, could you please provide your APISIX configuration file? What is your APISIX deployment structure? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
vortegatorres commented on issue #12769: URL: https://github.com/apache/apisix/issues/12769#issuecomment-3926352260 We have observed the same behaviour using [Helm chart 2.12.6](https://github.com/apache/apisix-helm-chart/tree/apisix-2.12.6). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
hebbaa commented on issue #12769:
URL: https://github.com/apache/apisix/issues/12769#issuecomment-3568844345
Our deployment is decoupled.
We have data-plane, ingress control plane, etcd.
Please find the overriden conf. A
deployment:
role: data_plane
role_data_plane:
config_provider: etcd
etcd:
host:
- "https://etcd-0.etcd-headless.apisix.svc.cluster.local:2379"
- "https://etcd-1.etcd-headless.apisix.svc.cluster.local:2379"
- "https://etcd-2.etcd-headless.apisix.svc.cluster.local:2379"
prefix: /apisix
tls:
cert: "/usr/local/apisix/certs/client.crt"
key: "/usr/local/apisix/certs/client.key"
nginx_config:
worker_processes: 4
worker_connections: 4096
worker_rlimit_nofile: 65535
http:
keepalive_timeout: "60s"
send_timeout: "300s"
proxy_connect_timeout: "90s"
proxy_send_timeout: "300s"
proxy_read_timeout: "300s"
access_log_format: |
{ "time": "$time_iso8601", "remote_addr": "$remote_addr", "request_id": "$request_id", "host": "$http_host", "request_uri": "$request_uri", "request_method": "$request_method", "status": "$status", "body_bytes_sent": "$body_bytes_sent", "request_time": "$request_tim
http_server_configuration_snippet: |
error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 421 422 423 424 425 426 428 429 431 451 500 501 502 503 504 505 506 507 508 510 511 @custom_error;
location @custom_error {
internal;
default_type text/html;
content_by_lua_block {
local status = ngx.status or 500
local messages = {
[400] = "Bad Request",
[401] = "Unauthorized",
[403] = "Forbidden",
[404] = "Not Found",
[500] = "Internal Server Error",
[502] = "Bad Gateway",
[503] = "Service Unavailable",
[504] = "Gateway Time-out"
}
local msg = messages[status] or "Unexpected Error"
ngx.say("<body><center><h1>" .. status .. " " .. msg .. "</h1></center></body>")
}
}
stream_configuration_snippet: |
upstream redis {
server apisix-redis.apisix.svc.cluster.local:6380;
}
server {
listen 127.0.0.1:6379;
proxy_pass redis;
proxy_ssl on;
proxy_ssl_certificate /usr/local/apisix/certs/redis-client.crt;
proxy_ssl_certificate_key /usr/local/apisix/certs/redis-client.key;
proxy_ssl_trusted_certificate /usr/local/apisix/certs/ca.crt;
}
apisix:
node_listen: 9080
enable_heartbeat: true
enable_report: false
enable_debug: false
enable_admin: false
extra_lua_path: /usr/local/apisix/custom_plugins/?.lua;
proxy_mode: "http&stream"
stream_proxy:
tcp:
- addr: "127.0.0.1:6379"
- addr: "0.0.0.0:8883"
- addr: "0.0.0.0:60000"
proxy_protocol:
listen_http_port: 9081
listen_https_port: 9082
ssl:
enable: true
listen:
- port: 9443
fallback_sni: <removed intentionally >
ssl_trusted_certificate: /usr/local/apisix/certs/ca.crt
ssl_ciphers: <removed intentionally >
plugins:
- prometheus
- fault-injection
- serverless-pre-function
- serverless-post-function
- file-logger
- redirect
- workflow
- mobiheaders-plugin
- mobiredirect-plugin
- mi-xsrf-token-filter
- request-id
- forward-auth
- mi-x509-details
- mi-auth-cookie-to-header
- mi-forward-by-cookie
- mi-forward-by-ddm-request
- mi-path-decoder-filter
- cors
- mi-signed-cookie-validate
- proxy-rewrite
- response-rewrite
plugin_attr:
prometheus: # Plugin: prometheus attributes
export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
enable_export_server: true # Enable the Prometheus export server.
export_addr: # Set the address for the Prometheus export server.
ip: 0.0.0.0 # Set the IP.
port: 9091
- conf/config.yaml 115/115 100%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
hebbaa commented on issue #12769: URL: https://github.com/apache/apisix/issues/12769#issuecomment-3568850252 The issue is because of OOM one of the worker got killed ``` dmesg | grep -i "killed" [1742140.134417] Memory cgroup out of memory: Killed process 1454261 (openresty) total-vm:637692kB, anon-rss:377044kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970 [1743469.967848] Memory cgroup out of memory: Killed process 1454264 (openresty) total-vm:637220kB, anon-rss:375624kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970 [1748701.318216] Memory cgroup out of memory: Killed process 1454262 (openresty) total-vm:636236kB, anon-rss:375532kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:896kB oom_score_adj:970 [1763367.552343] Memory cgroup out of memory: Killed process 198704 (openresty) total-vm:633328kB, anon-rss:373304kB, file-rss:10348kB, shmem-rss:20792kB, UID:636 pgtables:888kB oom_score_adj:970 ``` from apisix data-plane logs `"2025/11/23 14:29:08 [alert] 1#1: worker process 214 exited on signal 9". ` After the above looks like master is trying to start new worker process . During its startup the below errors seen `2025/11/23 10:46:52 [error] 54#54: *643711487 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer 2025/11/23 10:46:52 [error] 116#116: *643711540 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/events/compat/init.lua:47: attempt to index upvalue 'ev' (a nil value) 2025/11/23 10:46:52 [error] 52#52: *643711538 stream [lua] worker.lua:266: communicate(): event worker failed: failed to receive the header bytes: closed, context: ngx.timer` Issue to investigate is why the new worker process didn't start cleanly . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
hebbaa commented on issue #12769: URL: https://github.com/apache/apisix/issues/12769#issuecomment-3568182787 mem request and mem limit for the conatiner is set at 2GB -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
hebbaa commented on issue #12769: URL: https://github.com/apache/apisix/issues/12769#issuecomment-3574120269 No I have not configured . I have already mentioned the the cause for the issue. You can consider that as a step to reproduce. Simple step. 1. Make OOM error "[1742140.134417] Memory cgroup out of memory: Killed process 1454261 (openresty) total-vm:637692kB, anon-rss:377044kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970 " 2. master will kill the process 3. New process will not come up cleanly . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
hebbaa commented on issue #12769: URL: https://github.com/apache/apisix/issues/12769#issuecomment-3568148963 Looks like the above issue is occurring from killing of one or multiple worker nodes. dmesg | grep -i "killed" [1742140.134417] Memory cgroup out of memory: Killed process 1454261 (openresty) total-vm:637692kB, anon-rss:377044kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970 [1743469.967848] Memory cgroup out of memory: Killed process 1454264 (openresty) total-vm:637220kB, anon-rss:375624kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:904kB oom_score_adj:970 [1748701.318216] Memory cgroup out of memory: Killed process 1454262 (openresty) total-vm:636236kB, anon-rss:375532kB, file-rss:10320kB, shmem-rss:20776kB, UID:636 pgtables:896kB oom_score_adj:970 [1763367.552343] Memory cgroup out of memory: Killed process 198704 (openresty) total-vm:633328kB, anon-rss:373304kB, file-rss:10348kB, shmem-rss:20792kB, UID:636 pgtables:888kB oom_score_adj:970 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
Baoyuantop commented on issue #12769: URL: https://github.com/apache/apisix/issues/12769#issuecomment-4037322583 The two questions reported in this issue are expected side effects of an OpenResty worker process being killed by an OOM (Out of Memory) error: 1. "failed to receive the header bytes: closed" — After the broker worker is killed, the socket connections of other workers are broken. lua-resty-events has a built-in automatic reconnection mechanism, so this error is temporary and self-healing. 2. "attempt to index upvalue 'ev' (a nil value)" — A brief race condition during worker restart, where run() is called before configure() completes. This is a manifestation of a lack of defensive checks in the lua-resty-events compat layer, but the triggering condition is the OOM kill exception. The core issue is not a logical flaw in APISIX or lua-resty-events, but rather that insufficient container memory leads to an OOM kill. Refer to https://github.com/apache/apisix/pull/10550#discussion_r1419912672 , we will ignore these errors in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org