[I] opentelemetry plugin: uncaught Lua error on worker shutdown via vendored opentelemetry-lua batch_span_processor [apisix]

["yeganeahmadnejad (via GitHub)" <gi...@apache.org>]

yeganeahmadnejad opened a new issue, #13980:
URL: https://github.com/apache/apisix/issues/13980

   ## Description
   
   The vendored `opentelemetry-lua` dependency (pinned `opentelemetry-lua = 0.2-6` in `apisix-master-0.rockspec`) has a bug in `batch_span_processor.lua`'s `create_timer()`: when `ngx.timer.at()` fails because the worker is exiting -- expected near the end of every graceful shutdown/deploy -- it falls back to a *synchronous* span export (`self:flush_all()`, opening a cosocket).
   
   Since spans finish inline inside whatever phase the `opentelemetry` plugin is running (`body_filter_by_lua*` or `log_by_lua*` for HTTP spans -- see `apisix/plugins/opentelemetry.lua`'s `phase_func`), that synchronous cosocket call hits APISIX's own phase guard in `apisix/patch.lua` and crashes:
   
   ```
   2026/09/20 09:31:05 [error] 70#70: *1277095 failed to run log_by_lua*: /usr/local/apisix/apisix/patch.lua:406: API disabled in the context of log_by_lua*
   ```
   
   Full traceback (captured on 3.18.0):
   
   ```
   batch_span_processor.lua:97: create_timer(): failed to create timer: process exiting while logging request
   stack traceback:
   	[C]: in function 'error'
   	.../resty/core/socket/tcp.lua:298: in function 'original_tcp'
   	.../apisix/patch.lua:406: in function 'ngx_socket_tcp'
   	.../resty/http.lua:227: in function 'new'
   	.../opentelemetry/trace/exporter/http_client.lua:36: in function 'do_request'
   	.../opentelemetry/trace/exporter/otlp.lua:58: in function 'export_spans'
   	.../opentelemetry/trace/batch_span_processor.lua:45: in function 'process_batches'
   	.../opentelemetry/trace/batch_span_processor.lua:215: in function 'flush_all'
   	.../opentelemetry/trace/batch_span_processor.lua:99: in function 'create_timer'
   	.../opentelemetry/trace/batch_span_processor.lua:183: in function 'on_end'
   	.../opentelemetry/trace/recording_span.lua:98: in function 'finish'
   	/usr/local/apisix/apisix/plugins/opentelemetry.lua:559: in function 'phase_func'
   	/usr/local/apisix/apisix/plugin.lua:1442: in function 'common_phase'
   	/usr/local/apisix/apisix/init.lua:1213: in function 'http_log_phase'
   ```
   
   ## Steps to reproduce
   
   1. Enable the `opentelemetry` plugin with `tracing: true` (or the default per-route tracing) on any route -- including one with no other plugins, e.g. just a health-check route.
   2. Deploy/restart the worker (trigger a graceful shutdown) while traffic is hitting that route.
   3. Near the end of the outgoing worker's shutdown, the last span(s) to finish will hit the race and crash with the above error.
   
   Because this only requires the worker to be exiting -- not any specific plugin config -- it fires ~consistently on every rolling deploy of any release with tracing enabled.
   
   ## Root cause & fix
   
   Filed and fixed upstream in opentelemetry-lua: yangxikun/opentelemetry-lua#107, fix at yangxikun/opentelemetry-lua#106. Once merged/released there, bumping the pinned `opentelemetry-lua` version here would pick up the fix. In the meantime, APISIX could consider wrapping `phase_func`'s span-finishing call in a `pcall` as defense in depth, since this class of "third-party tracer code doing something disallowed in a restricted phase" isn't fully preventable from the library side alone.
   
   ## Environment
   
   - APISIX 3.18.0 (Ubuntu-based image), `opentelemetry-lua = 0.2-6`
   - OpenResty/CentOS or Ubuntu base, not relevant to the phase-restriction behavior (standard OpenResty `body_filter_by_lua*`/`log_by_lua*` cosocket restriction)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org