You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "aglinxinyuan (via GitHub)" <gi...@apache.org> on 2026/07/22 10:45:20 UTC
[I] Reduce CI log verbosity from per-worker/per-message amber engine logs [texera]
aglinxinyuan opened a new issue, #6796:
URL: https://github.com/apache/texera/issues/6796
### Problem
CI logs — especially the amber integration specs (e.g. `LoopIntegrationSpec`) — are flooded with per-worker / per-message engine logs at `INFO`. A single loop run emits thousands of lines: two full `ChannelIdentity` dumps per embedded control message (ECM), plus per-worker lifecycle and per-region lines, repeated on **every loop iteration** because workers are recreated each iteration. This bloats log storage and scroll, slows the run, and buries genuine failures.
### Root cause
`AmberLogging` names each logger `"<actorId>] [<ClassName>"` so the logback pattern `[%logger]` can render both the actor id and the class in one field. The side effect: these logger names start with the **actor id** (e.g. `WF6-…`, `COORDINATOR`), not `org.apache…`, so the
```xml
<logger name="org.apache" level="WARN"/>
```
rule in `amber/src/main/resources/logback.xml` never matches the engine classes. They all fall through to the root logger (`${TEXERA_SERVICE_LOG_LEVEL:-INFO}`) and emit at `INFO`. Lowering the `org.apache` level does nothing for them.
### Noise categories
| Volume | Message | Level | Source |
|---|---|---|---|
| Huge (3×/ECM, full URI dump) | `receive` / `process` / `send ECM` | INFO | `DataProcessor.scala`, `main_loop.py` |
| High | `register <id> -> Actor[…]` | INFO | `PekkoActorRefMappingService.scala` |
| High (misleading) | `…is not reachable anymore, it might have crashed` — fires on **normal** teardown | WARN | `PekkoActorRefMappingService.scala` |
| High (race noise) | `unknown identifier` | WARN | `PekkoActorRefMappingService.scala` |
| Per worker | `worker replay log writing conf`, `DP thread started`/`exits`, `Starting the worker.` | INFO | `WorkflowActor`, `DPThread`, `StartHandler` |
| Per region | `Region N successfully terminated.` | INFO | `RegionExecutionManager.scala` |
| Per operator | `…completed, # of input ports…` | INFO | `DataProcessor.scala` |
| Per client msg | `client actor cannot handle …` | WARN | `ClientActor.scala` |
### Fix (this task)
1. Demote the chatty per-message / per-worker-lifecycle / per-region logs to `DEBUG` so they no longer print at the default `INFO` level (CI, local, and prod). This includes the three misleading `WARN`s that fire on *normal* operation (`might have crashed` on graceful teardown, `unknown identifier` registration race, `client actor cannot handle` fallthrough). Genuine fault paths are left untouched (`DP Thread exists unexpectedly` ERROR, `Failed to fetch actorRef` WARN, `Error when terminating region` WARN).
2. CI backstop: pin `TEXERA_SERVICE_LOG_LEVEL=WARN` and `UDF_PYTHON_LOG_STREAMHANDLER_LEVEL=WARN` on the amber unit and integration test steps in `build.yml`.
### Follow-up candidates (not in the first PR)
An engine-wide sweep surfaced these additional hot-path logs that could be demoted in a later pass:
| Location | Level | Why noisy |
|---|---|---|
| `AsyncRPCClient.scala` (`receive reply: null …`) | INFO | Per control-reply; inconsistent sibling of the `debug` branch a few lines above |
| `range_based_shuffle_partitioner.py` (`got {partitioning}`) | INFO | Per-worker partition setup; outlier — no other partitioner logs at INFO in its constructor |
| `tuple.py` (out-of-range / cast warnings) | WARNING | Data-conditional per-tuple/per-field; can flood on a single bad column. Genuinely useful, so consider once-per-column dedupe rather than demotion |
| `ClientActor.scala` (`Amber Client received: …`) | INFO | Per-message fallback; low confidence it's a real flood |
Latent test note: `PekkoConfigSpec` asserts `pekko.stdout-loglevel == "INFO"` **unguarded**. It is safe today only because `stdout-loglevel` is hardcoded in `cluster.conf`. If a `${?TEXERA_SERVICE_LOG_LEVEL}` override is ever added to that key, the assertion will start failing in the CI step that now sets `WARN` — either move it inside the `sys.env`/`sys.props` guard or add a comment noting the dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce CI log verbosity from per-worker/per-message amber engine logs [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6796:
URL: https://github.com/apache/texera/issues/6796#issuecomment-5045173789
<!-- texera:template-compliance -->
👋 Thanks for opening this issue, @aglinxinyuan!
It looks like the issue description doesn't quite follow our template yet:
- The **Task Summary** section is missing; please keep the template's headings.
Filling out the template helps reviewers understand and triage your contribution faster. Please edit the description to complete it. This message will disappear automatically once the template is followed.
You can find the template prompts by editing the description, or see [CONTRIBUTING.md](https://github.com/apache/texera/blob/main/CONTRIBUTING.md) for the full contribution flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Reduce CI log verbosity from per-worker/per-message amber engine logs [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan closed issue #6796: Reduce CI log verbosity from per-worker/per-message amber engine logs
URL: https://github.com/apache/texera/issues/6796
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org