You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/07/27 03:58:22 UTC
[I] Successful region teardown logs alarming per-worker WARNs ("might have crashed", "unknown identifier") [texera]
aglinxinyuan opened a new issue, #6922:
URL: https://github.com/apache/texera/issues/6922
### What happened?
Every **successful** region teardown logs a batch of alarming WARNs that read like crashes, scaling with worker count:
| log line | where | when it fires on a healthy teardown |
|---|---|---|
| `actor X is not reachable anymore, it might have crashed. old ref = ...` | `PekkoActorRefMappingService.scala:76-81` (`removeActorRef`) | once **per worker** — `RegionExecutionManager.scala:198` calls `removeActorRef` unconditionally for every worker of every completed region |
| same WARN again | via the dead-letter branch `WorkflowActor.scala` (`MessageBecomesDeadLetter` → `removeActorRef(dest)`) | again from every peer that still had an in-flight message to the stopped worker |
| `unknown identifier: ...` | `PekkoActorRefMappingService.scala:122` | on every send to a worker that was already removed (e.g. resent messages, late stats queries) |
So an N-worker region emits ≥N "might have crashed" WARNs on the *normal* completion path — before any real problem exists. This trains operators (and CI log readers) to ignore WARNs, and it buries genuine crash evidence. It is the largest remaining contributor to teardown log noise now that the EndWorker retry storm is fixed (#6916).
Note the logger names here are Amber-style (`<actorId>] [<Class>`), which never match package-prefix logback filters like `org.apache=WARN`, so this cannot be silenced by logback configuration — the call sites themselves need to distinguish "removed deliberately during teardown" from "vanished unexpectedly" (e.g. a `removeActorRef(id, expected = true)` variant logged at debug/info).
### How to reproduce?
Run any workflow to completion and grep the coordinator log for `might have crashed` — one WARN per worker per region despite a fully successful run. `LoopIntegrationSpec` on CI shows this repeatedly (once per loop iteration per region).
### Version/Branch
main (observed at 429be110a7; related to the log-noise follow-ups in #6796 and the fix in #6916).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Successful region teardown logs alarming per-worker WARNs ("might have crashed", "unknown identifier") [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan closed issue #6922: Successful region teardown logs alarming per-worker WARNs ("might have crashed", "unknown identifier")
URL: https://github.com/apache/texera/issues/6922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Successful region teardown logs alarming per-worker WARNs ("might have crashed", "unknown identifier") [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on issue #6922:
URL: https://github.com/apache/texera/issues/6922#issuecomment-5099239063
fixed in #6797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Successful region teardown logs alarming per-worker WARNs ("might have crashed", "unknown identifier") [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6922:
URL: https://github.com/apache/texera/issues/6922#issuecomment-5087106420
<!-- texera:template-compliance -->
👋 Thanks for opening this issue, @aglinxinyuan!
It looks like the issue description doesn't quite follow our template yet:
- This issue doesn't appear to use one of our templates (Bug, Feature, or Task). Please open it using a template so the required details are captured.
Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed.
You can find the template prompts by editing the description, or see [CONTRIBUTING.md](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) for the full contribution flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org