You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hugegraph.apache.org by "imbajin (via GitHub)" <gi...@apache.org> on 2026/06/21 13:27:35 UTC
[I] [Task] Investigate slow integrate-test stalls in message sender barrier [hugegraph-computer]
imbajin opened a new issue, #350:
URL: https://github.com/apache/hugegraph-computer/issues/350
## Background
The `Integrate test` job for HugeGraph Computer can take a long time even though the integration suite is small. The slow part is usually not the number of tests, but a long wait in the message/input synchronization path.
An observed CI log repeatedly prints:
```text
EtcdClient - Wait for keys with prefix 'BSP_WORKER_INPUT_DONE' and timeout 86400000ms, expect 1 keys but actual got 0 keys
```
The same log shows the worker entering input step and starting vertex message sending before the wait:
```text
WorkerService inputstep started
MessageSendManager - Start sending message(type=VERTEX)
```
So the master is waiting for the worker's `BSP_WORKER_INPUT_DONE` signal, but the worker has not reached `Bsp4Worker.workerInputDone()` yet.
## Initial code pointers
- CI runs `mvn test -P integrate-test -ntp` in `.github/workflows/computer-ci.yml`.
- The `integrate-test` profile includes `IntegrateTestSuite`, which currently contains `SenderIntegrateTest`.
- `SenderIntegrateTest` has only a few cases, but `testOneWorkerWithBusyClient()` intentionally slows the send path by wrapping the client's send function with `Thread.sleep(100)`.
- `WorkerInputManager.loadGraph()` sends vertices and edges first. Only after it returns does `WorkerService.inputstep()` call `bsp4Worker.workerInputDone()`.
- `ComputerOptions.BSP_WAIT_WORKERS_TIMEOUT` and `BSP_WAIT_MASTER_TIMEOUT` default to 24 hours, so a hidden sender/session/input problem can become a very slow CI wait instead of a fast, actionable failure.
Relevant files:
```text
.github/workflows/computer-ci.yml
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/IntegrateTestSuite.java
computer/computer-test/src/main/java/org/apache/hugegraph/computer/suite/integrate/SenderIntegrateTest.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/input/WorkerInputManager.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/worker/WorkerService.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/QueuedMessageSender.java
computer/computer-core/src/main/java/org/apache/hugegraph/computer/core/sender/MessageSendManager.java
computer/computer-api/src/main/java/org/apache/hugegraph/computer/core/config/ComputerOptions.java
```
Related prior symptom: #203 reported `The origin future must be null` in `SenderIntegrateTest`. That may be in the same control-message/future/session area, but this task is specifically about the slow CI wait and fail-fast/debuggability of the integration test.
## Suggested investigation
1. Reproduce the integration suite with etcd available:
```bash
cd computer
mvn test -P integrate-test -Dtest=IntegrateTestSuite -ntp
```
2. Confirm which test case spends time before `BSP_WORKER_INPUT_DONE`. Start with `SenderIntegrateTest#testOneWorkerWithBusyClient`.
3. Trace the input path:
```text
SenderIntegrateTest
-> WorkerService.execute()
-> WorkerService.inputstep()
-> WorkerInputManager.loadGraph()
-> MessageSendManager.startSend()/finishSend()
-> QueuedMessageSender.send()
-> Bsp4Worker.workerInputDone()
```
4. Check whether START/FINISH control futures in `QueuedMessageSender` can be left stale, completed late, or hidden behind the sender thread. The old #203 stack around `futureRef` is a useful clue.
5. Make the test fail fast and print useful diagnostics. Possible directions:
- set much smaller `bsp.wait_workers_timeout` / `bsp.wait_master_timeout` for integration tests;
- add a JUnit/test-level timeout around each integration case;
- dump worker/master thread states when the input barrier is not reached;
- ensure sender exceptions propagate to both the worker future and the master-side wait;
- replace the sleep-based busy-client simulation with a more deterministic back-pressure or blocked-client fixture.
## Expected result
- Integration tests should not spend many minutes printing only `BSP_WORKER_INPUT_DONE` wait logs.
- If the sender/input path is broken, the test should fail quickly with an actionable error and enough thread/session state to locate the failing component.
- The slow/busy-client path should have regression coverage so future changes do not reintroduce the long wait.
## Newcomer scope
This is a good newcomer task because the suspected area is narrow: one integration suite, the input-step barrier, and the message sender control future path. A complete fix does not need a large algorithm or distributed-runtime redesign; first improving timeout/diagnostics and then isolating the sender/session condition would already be valuable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [I] [Task] Investigate slow integrate-test stalls in message sender barrier [hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin closed issue #350: [Task] Investigate slow integrate-test stalls in message sender barrier
URL: https://github.com/apache/hugegraph-computer/issues/350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [I] [Task] Investigate slow integrate-test stalls in message sender barrier [hugegraph-computer]
Posted by "wipheg (via GitHub)" <gi...@apache.org>.
wipheg commented on issue #350:
URL: https://github.com/apache/hugegraph-computer/issues/350#issuecomment-4762544176
Hi @imbajin,
I'd like to take a look at this issue if no one is working on it yet.
I'll start by reproducing the integration test locally and checking where the worker gets stuck before reaching `BSP_WORKER_INPUT_DONE`, especially around the sender/input path mentioned in the description.
Once I have some findings, I'll post an update here.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [I] [Task] Investigate slow integrate-test stalls in message sender barrier [hugegraph-computer]
Posted by "PhoenixRushi (via GitHub)" <gi...@apache.org>.
PhoenixRushi commented on issue #350:
URL: https://github.com/apache/hugegraph-computer/issues/350#issuecomment-4765766211
Hi @imbajin,
I'd like to investigate this issue.
From my initial review, it seems that the worker enters the input/message-sending flow but may not reach `workerInputDone()`, which causes the master to keep waiting for `BSP_WORKER_INPUT_DONE.
I'll try to reproduce the integration test locally and trace the execution path around `WorkerInputManager`, `MessageSendManager`, and `QueuedMessageSender` to identify where the flow is getting blocked.
I'll share my findings here as I dig deeper.
Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [I] [Task] Investigate slow integrate-test stalls in message sender barrier [hugegraph-computer]
Posted by "PhoenixRushi (via GitHub)" <gi...@apache.org>.
PhoenixRushi commented on issue #350:
URL: https://github.com/apache/hugegraph-computer/issues/350#issuecomment-4767157351
Thanks for the clarification.
I'll start by reproducing the issue locally and gathering the requested details, including the exact test behavior and logs around `WorkerInputManager`, `MessageSendManager`, and `QueuedMessageSender`.
I'll post my findings here as I progress so we can avoid overlapping work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org
Re: [I] [Task] Investigate slow integrate-test stalls in message sender barrier [hugegraph-computer]
Posted by "imbajin (via GitHub)" <gi...@apache.org>.
imbajin commented on issue #350:
URL: https://github.com/apache/hugegraph-computer/issues/350#issuecomment-4766073511
@wipheg @PhoenixRushi Thanks both for your interest.
This issue is mainly an investigation task, so it would be helpful if both of you
can take a look and share findings here.
To avoid duplicated work, please post short updates when you have:
- a reproducible command/environment;
- the exact test case that stalls;
- thread/session logs around `WorkerInputManager`, `MessageSendManager`, or
`QueuedMessageSender`;
- a suspected root cause.
Once one of you has a clear root cause and a fix direction, feel free to open a
PR and link it here. The other person is very welcome to help verify the PR or
review the diagnosis.
Please coordinate in this thread before opening overlapping PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@hugegraph.apache.org
For additional commands, e-mail: issues-help@hugegraph.apache.org