[I] bug: healthcheck not work [apisix]

["elizax (via GitHub)" <gi...@apache.org>]

elizax opened a new issue, #13235:
URL: https://github.com/apache/apisix/issues/13235

   ### Current Behavior
   
   # 健康检查失效 Bug 修复总结
   
   ## 问题描述
   
   在 Apisix API 网关中，上游健康检查（upstream health check）在以下场景中会失效：
   
   1. **重启服务后** - 健康检查工作正常
   2. **禁用健康检查** - 健康检查停止
   3. **重新启用健康检查** - 健康检查**失效**，不再发送主动探测请求
   
   ### 失效表现
   
   - 后端服务收不到健康检查请求
   - 网关日志显示定时器空转，不执行实际检查
   - 即使配置正确，健康检查也无法恢复
   
   ## 问题根因分析
   
   ### 技术背景
   
   Apisix 使用 `resty.healthcheck` 模块进行健康检查，采用多 worker 架构：
   
   - 每个 worker 有独立的 `hcs` 表（存储 checker 实例）
   - 使用共享内存（SHM）中的周期性锁（`PERIODIC_LOCK`）协调多个 worker
   - 只有一个 worker 能持有锁并执行健康检查
   
   ### 根因
   
   当禁用/重新启用健康检查时，发生以下问题：
   
   1. **禁用时**：Worker 2 的 checker 被标记为 `dead=true`，`healthy.active=false`，但定时器仍在运行并持有锁
   2. **重新启用时**：Worker 1 创建了新 checker（`healthy.active=true`），但无法获取锁
   3. **结果**：Worker 1 的新 checker 无法执行检查，Worker 2 的旧 checker 虽然持有锁但不执行检查（因为 `active=false`）
   
   ### 日志证据
   
   ```
   # Worker 3 持有锁但无活跃 checker
   worker 3 (pid 1067941): [HC-DEBUG] hcs[1] healthy.active=false unhealthy.active=false
   
   # Worker 1 有活跃 checker 但无法获取锁
   worker 1 (pid 1067939): [TIMER-EXPIRE] get_periodic_lock failed, skipping this iteration
   worker 1 (pid 1067939): [HC-DEBUG] hcs[1] healthy.active=true unhealthy.active=true
   ```
   
   ## 修复方案
   
   ### 修改文件
   
   1. `Apisix/deps/share/lua/5.1/resty/healthcheck.lua`
   2. `Apisix/Apisix/healthcheck_manager.lua`
   
   ### 核心修改
   
   #### 1. healthcheck.lua - 定时器循环改进
   
   在定时器获取锁后，添加活跃 checker 检查：
   
   ```lua
   -- 检查是否有活跃的 checker，如果没有则释放锁并退出
   local has_active_checker = false
   for _, checker_obj in ipairs(hcs) do
     if not checker_obj.dead and
        (checker_obj.checks.active.healthy.active or
         checker_obj.checks.active.unhealthy.active) then
       has_active_checker = true
       break
     end
   end
   if not has_active_checker then
     -- 没有活跃的 checker，释放锁并让定时器进入休眠
     shm:delete(key)
     active_check_timer.interval = CHECK_INTERVAL * 10
     return
   end
   ```
   
   在遍历 `hcs` 表时，跳过 dead 的 checker：
   
   ```lua
   for _, checker_obj in pairs(hcs) do
     -- 跳过 dead 的 checker
     if checker_obj.dead then
       goto continue_checker
     end
     -- ... 执行健康检查逻辑
     ::continue_checker::
   end
   ```
   
   #### 2. healthcheck_manager.lua - 清理调试日志
   
   移除调试期间添加的日志输出，保持代码简洁。
   
   ## 修复验证
   
   ### 测试场景
   
   1. ✅ 重启服务 - 健康检查正常工作
   2. ✅ 禁用健康检查 - 锁被正确释放
   3. ✅ 重新启用健康检查 - 健康检查恢复正常
   
   ### 正常日志示例
   
   ```
   # Worker 1 成功执行健康检查
   worker 1 (pid 1067939): [TIMER-CALLBACK] running healthy check for: upstream#/Apisix/routes/xxx
   worker 1 (pid 1067939): [TIMER-CALLBACK] running unhealthy check for: upstream#/Apisix/routes/xxx
   
   # Worker 3 正确放弃锁
   worker 3 (pid 1067941): [TIMER-EXPIRE] get_periodic_lock failed, skipping this iteration
   ```
   
   ## 技术要点
   
   ### 多 Worker 锁机制
   
   - `PERIODIC_LOCK` 是全局共享锁，所有 checker 共享
   - 锁持有者必须定期更新锁的过期时间（`LOCK_PERIOD = CHECK_INTERVAL * 15`）
   - 其他 worker 无法获取锁时会降低定时器频率（`CHECK_INTERVAL * 10`）
   
   ### 关键设计
   
   1. **锁释放时机**：当没有活跃 checker 时，必须立即释放锁
   2. **Dead Checker 处理**：dead checker 不应参与健康检查循环
   3. **优雅降级**：无法获取锁时，定时器进入低频运行状态
   
   ## 相关文件
   
   - `apisix/deps/share/lua/5.1/resty/healthcheck.lua` - 健康检查核心模块
   - `apisix/apisix/healthcheck_manager.lua` - 健康检查管理器
   
   ## 修复日期
   
   2026-04-17
   
   ### Expected Behavior
   
   _No response_
   
   ### Error Logs
   
   _No response_
   
   ### Steps to Reproduce
   
   重启服务后 - 健康检查工作正常
   禁用健康检查 - 健康检查停止
   重新启用健康检查 - 健康检查失效，不再发送主动探测请求
   
   ### Environment
   
   - APISIX version (run `apisix version`):3.14.1
   - Operating system (run `uname -a`):centos
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug: healthcheck not work [apisix]

["AlinsRan (via GitHub)" <gi...@apache.org>]

AlinsRan closed issue #13235: bug: healthcheck not work
URL: https://github.com/apache/apisix/issues/13235


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug: healthcheck not work [apisix]

["Baoyuantop (via GitHub)" <gi...@apache.org>]

Baoyuantop commented on issue #13235:
URL: https://github.com/apache/apisix/issues/13235#issuecomment-4265181889

   Hi @elizax, please use English in public channels. I've created a PR in the upstream repository. https://github.com/api7/lua-resty-healthcheck/pull/54


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug: healthcheck not work [apisix]

["elizax (via GitHub)" <gi...@apache.org>]

elizax commented on issue #13235:
URL: https://github.com/apache/apisix/issues/13235#issuecomment-4264967520

   I fixed the problems I encountered during the use with AI and tested that the functions are working fine. Here's the summary. You can take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org