You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@superset.apache.org by jl...@apache.org on 2026/09/27 14:11:12 UTC
(superset) branch ultraset-mh-databend-grpc-2 created (now 5907a327d97)
This is an automated email from the ASF dual-hosted git repository.
sadpandajoe pushed a change to branch ultraset-mh-databend-grpc-2
in repository https://gitbox.apache.org/repos/asf/superset.git
at 5907a327d97 fix(ci): retry databend testcontainers bring-up past the image's metasrv race
This branch includes the following new commits:
new 5907a327d97 fix(ci): retry databend testcontainers bring-up past the image's metasrv race
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(superset) 01/01: fix(ci): retry databend testcontainers bring-up past the image's metasrv race
Posted by jl...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
sadpandajoe pushed a commit to branch ultraset-mh-databend-grpc-2
in repository https://gitbox.apache.org/repos/asf/superset.git
commit 5907a327d9762cba730881450b135eb33fefc584
Author: Joe Li <47...@users.noreply.github.com>
AuthorDate: Sun Sep 27 14:10:53 2026 +0000
fix(ci): retry databend testcontainers bring-up past the image's metasrv race
The `testcontainers (databend, 10)` job fails intermittently on master --
most recently in run 36296832911 (2026-09-27, master @ eb2ffa28cc), and
before that in run 34816535396 (2026-09-14) -- with:
TimeoutError: Container did not emit logs containing
'listened at 0.0.0.0:8000' within 120.000 seconds.
Container status: running
The cause is in the image's own entrypoint, not in Superset. In
databendlabs/databend `docker/bootstrap.sh` (identical at the failing build
`v1.2.881-ca29960f5c-simd`, at tag v1.2.881 and on main), the entire
metasrv readiness gate is a bare `sleep 1`:
databend-meta ... --single &>/tmp/std-meta.log &
PROCESSES+=($!)
# wait for meta to be ready
sleep 1
...
databend-query -c "$QUERY_CONFIG_FILE" &>/tmp/std-query.log &
PROCESSES+=($!)
...
wait "${PROCESSES[@]}"
Before metasrv binds its gRPC port it must finish an on-disk data upgrade
(V003 -> V004, including sled-tree removal) and single-node leader
election. When that overruns the one second -- occasionally, on a loaded
runner -- databend-query's single connection attempt is refused
(`Connection refused (os error 111)`), query exits, and nothing restarts
it. The closing `wait` keeps waiting on the surviving metasrv and log-tail
PIDs, so the container reports `running` forever while the banner line the
wait strategy looks for can never be emitted.
That rules out widening the wait timeout the way test_db2.py legitimately
needs to: no timeout can make a dead one-shot process re-emit a line. Only
a fresh container can re-run the race, so bring-up is retried up to three
times, stopping each container it gives up on and re-raising the original
TimeoutError once the attempts are exhausted.
The durable fix belongs upstream (poll the gRPC port instead of sleeping,
and supervise databend-query rather than letting `wait` mask its death);
this keeps master green in the meantime.
Co-Authored-By: Claude Opus 5 <no...@anthropic.com>
---
.../db_engine_specs/test_databend.py | 37 ++++++++++++++++++----
1 file changed, 30 insertions(+), 7 deletions(-)
diff --git a/tests/testcontainers/db_engine_specs/test_databend.py b/tests/testcontainers/db_engine_specs/test_databend.py
index e4f4f1f7662..21b602ca134 100644
--- a/tests/testcontainers/db_engine_specs/test_databend.py
+++ b/tests/testcontainers/db_engine_specs/test_databend.py
@@ -59,22 +59,45 @@ from ._pagination import ( # noqa: E402
HTTP_PORT = 8000
DBNAME = "default"
+# The image's entrypoint (databendlabs/databend docker/bootstrap.sh) gates
+# databend-query on a bare `sleep 1` after backgrounding databend-meta. When
+# metasrv's on-disk data upgrade and leader election overrun that one second
+# -- intermittently, on a loaded runner -- query's single connection attempt
+# to metasrv's gRPC port is refused, query exits, and nothing restarts it.
+# The entrypoint's closing `wait` keeps waiting on the surviving metasrv and
+# log-tail PIDs, so the container stays "running" while the banner line below
+# can never be emitted. Widening the wait timeout therefore cannot help; only
+# a fresh container can re-run the race, hence retrying bring-up.
+START_ATTEMPTS = 3
+
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
- container = DockerContainer("datafuselabs/databend")
- container.with_exposed_ports(HTTP_PORT)
- # The image's own startup banner documents this exact line as proof its
- # HTTP query endpoint is bound and ready.
- container.waiting_for(LogMessageWaitStrategy(f"listened at 0.0.0.0:{HTTP_PORT}"))
-
- with container:
+ for attempt in range(1, START_ATTEMPTS + 1):
+ container = DockerContainer("datafuselabs/databend")
+ container.with_exposed_ports(HTTP_PORT)
+ # The image's own startup banner documents this exact line as proof its
+ # HTTP query endpoint is bound and ready.
+ container.waiting_for(
+ LogMessageWaitStrategy(f"listened at 0.0.0.0:{HTTP_PORT}")
+ )
+ try:
+ container.start()
+ break
+ except TimeoutError:
+ container.stop()
+ if attempt == START_ATTEMPTS:
+ raise
+
+ try:
host = container.get_container_host_ip()
port = container.get_exposed_port(HTTP_PORT)
# "root" with no password is the image's builtin user -- confirmed
# directly against a running container, not from the image's own
# doc text, which only shows ${USER}/${PASSWORD} placeholders.
yield create_engine(f"databend://root:@{host}:{port}/{DBNAME}?sslmode=disable")
+ finally:
+ container.stop()
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None: