You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "nic-6443 (via GitHub)" <gi...@apache.org> on 2026/04/18 03:53:58 UTC

[PR] ci: wait for stream subsystem readiness in test_prometheus_stream.sh [apisix]

nic-6443 opened a new pull request, #13253:
URL: https://github.com/apache/apisix/pull/13253

   ### Description
   
   `t/cli/test_prometheus_stream.sh` is flaky on the CI: many recent runs on `master` and feature branches fail with `failed: prometheus can't work in stream subsystem`. A few examples just from the last few days:
   
   - master `41775ef5` — https://github.com/apache/apisix/actions/runs/24496623331
   - master `ddddeafc` — https://github.com/apache/apisix/actions/runs/24488366032
   - branch `feat-support-bedrock` `8bf841b4` — https://github.com/apache/apisix/actions/runs/24561304700
   
   ### Root cause
   
   The script does:
   
   ```bash
   make run
   sleep 0.5
   # ...
   curl http://127.0.0.1:9100 || true   # probe stream proxy
   sleep 1
   out="$(curl http://127.0.0.1:9091/apisix/prometheus/metrics)"
   grep "apisix_stream_connection_total{route=\"1\"} 1"
   ```
   
   On slower GitHub Actions runners, 0.5s is not enough for the stream subsystem's prometheus exporter to come up after `make run`. The probe to port 9100 hits a half-initialized listener and gets `Connection reset by peer`, so the connection counter is never bumped, and the metric grep fails. Confirmed by inspecting the failure logs:
   
   - HELP/TYPE lines for `apisix_stream_connection_total` are absent from the metrics output in the failing runs (they're present in the passing runs).
   
   ### Fix
   
   Replace the fixed `sleep 0.5` with a polling loop (up to ~10s) that waits until the prometheus metrics endpoint exposes the stream metric's HELP line, which is a reliable signal that the stream prometheus plugin is loaded and ready to record traffic.
   
   The downstream `curl ... :9100; sleep 1` is left as-is — once the plugin is loaded, that single probe + 1s flush window already works (the metric has `refresh_interval: 1`).
   
   ### Checklist
   
   - [x] I have explained the need for this PR and the problem it solves
   - [x] I have explained the changes or the new features added to this PR
   - [x] I have added tests corresponding to the changes introduced in this PR — N/A, this PR fixes an existing test
   - [x] I have added proper labels to this PR
   - [x] Review and CI completed successfully


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ci: wait for stream subsystem readiness in test_prometheus_stream.sh [apisix]

Posted by "nic-6443 (via GitHub)" <gi...@apache.org>.
nic-6443 closed pull request #13253: ci: wait for stream subsystem readiness in test_prometheus_stream.sh
URL: https://github.com/apache/apisix/pull/13253


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] ci: wait for stream subsystem readiness in test_prometheus_stream.sh [apisix]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13253:
URL: https://github.com/apache/apisix/pull/13253#discussion_r3104447291


##########
t/cli/test_prometheus_stream.sh:
##########
@@ -83,7 +94,16 @@ plugin_attr:
 " > conf/config.yaml
 
 make run
-sleep 0.5
+
+# Same readiness wait as above, since admin is disabled here we can't probe
+# admin API; the prometheus HELP line is still a reliable readiness signal.
+for _ in $(seq 1 20); do
+    if curl -s --max-time 2 http://127.0.0.1:9091/apisix/prometheus/metrics \
+            2>/dev/null | grep -q "# HELP apisix_stream_connection_total"; then
+        break
+    fi
+    sleep 0.5
+done

Review Comment:
   Same as the earlier readiness loop: if the HELP line never appears, the loop falls through and the test continues with a generic failure later. Consider explicitly failing after the loop (with a targeted message) when readiness wasn’t reached within the polling budget.



##########
t/cli/test_prometheus_stream.sh:
##########
@@ -83,7 +94,16 @@ plugin_attr:
 " > conf/config.yaml
 
 make run
-sleep 0.5
+
+# Same readiness wait as above, since admin is disabled here we can't probe
+# admin API; the prometheus HELP line is still a reliable readiness signal.
+for _ in $(seq 1 20); do
+    if curl -s --max-time 2 http://127.0.0.1:9091/apisix/prometheus/metrics \
+            2>/dev/null | grep -q "# HELP apisix_stream_connection_total"; then
+        break
+    fi
+    sleep 0.5
+done

Review Comment:
   The readiness-wait logic is duplicated in two places. To avoid the two copies drifting over time (e.g., if the timeout/tuning changes), consider extracting it into a small shell function (e.g., `wait_for_stream_prometheus_ready`) and calling it from both sections.



##########
t/cli/test_prometheus_stream.sh:
##########
@@ -36,7 +36,18 @@ plugin_attr:
 " > conf/config.yaml
 
 make run
-sleep 0.5
+
+# Wait until the stream subsystem's prometheus exporter is ready by polling
+# for the metric's HELP line, which appears once the stream plugin is loaded.
+# Without this, the test is flaky on slow CI runners where `sleep 0.5` is not
+# enough for the stream subsystem to come up after `make run`.
+for _ in $(seq 1 20); do
+    if curl -s --max-time 2 http://127.0.0.1:9091/apisix/prometheus/metrics \
+            2>/dev/null | grep -q "# HELP apisix_stream_connection_total"; then
+        break
+    fi
+    sleep 0.5
+done
 

Review Comment:
   The polling loop can silently time out and continue, which can lead to a later failure (or even a hang on the subsequent metrics `curl`) with a less actionable error. Consider tracking whether readiness was observed and explicitly `exit 1` with a clear message (optionally printing the last metrics response) when the loop completes without finding the HELP line.



##########
t/cli/test_prometheus_stream.sh:
##########
@@ -36,7 +36,18 @@ plugin_attr:
 " > conf/config.yaml
 
 make run
-sleep 0.5
+
+# Wait until the stream subsystem's prometheus exporter is ready by polling
+# for the metric's HELP line, which appears once the stream plugin is loaded.
+# Without this, the test is flaky on slow CI runners where `sleep 0.5` is not
+# enough for the stream subsystem to come up after `make run`.
+for _ in $(seq 1 20); do
+    if curl -s --max-time 2 http://127.0.0.1:9091/apisix/prometheus/metrics \
+            2>/dev/null | grep -q "# HELP apisix_stream_connection_total"; then
+        break
+    fi
+    sleep 0.5

Review Comment:
   This loop’s worst-case runtime is higher than it looks because each iteration allows `curl --max-time 2` plus the `sleep 0.5` (up to ~50s total). If the intent is ~10s, consider using a small `--connect-timeout` (and a lower `--max-time`) so failures return quickly while still polling for readiness.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org