You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "corgy-w (via GitHub)" <gi...@apache.org> on 2026/08/08 05:19:58 UTC
[PR] [Feature][Zeta] Prepare job log mode context and exact filename matching [seatunnel]
corgy-w opened a new pull request, #11702:
URL: https://github.com/apache/seatunnel/pull/11702
### Purpose of this pull request
This PR backports Phase A infrastructure changes from the internal ST-3605 work to the open-source Zeta engine so that the engine-server side is ready for future retention policy work without depending on whaletunnel-private classes.
Concretely:
- `MDCContext` gains an optional `JobMode` dimension. The legacy 3-segment wire format (`jobId/pipelineId/taskId`) stays compatible; the new 4-segment format (`jobId/pipelineId/taskId/mode") carries the job mode. When a nested context activates with a null mode, the previous `ST-JOB-MODE` MDC key is explicitly removed so cleanup policy state cannot leak across activations.
- `CoordinatorService` now builds job-scoped MDC executors via a new helper that pulls the job mode out of `JobImmutableInformation` (fail closed to unclassified when the payload is incomplete or corrupted) and threads it through every job executor entry point: `submitJob`, the master-node-switch restore path, the pending-job schedule loop, and the per-job `restoreJobFromMasterActiveSwitch` path.
- New `JobLogFileNameMatcher` performs exact matching of `job-<id>.log`, `job-<id>.log.unclassified`, and `job-<id>.log.yyyy-MM-dd-index` filenames. Adjacent job ids (e.g. `job-1234.log` for `jobId=123`) and unsupported suffixes (`.tmp`, bare date) are now rejected, fixing the historical prefix-substring download/cleanup leak. A separate `isRolledSegment` helper exists for the pruner so it can never touch active or unclassified files.
- `TaskLogManagerService` is split into two lifecycles:
- `clean(jobId)` keeps terminal removal. It is still invoked from `JobHistoryService.JobInfoExpiredListener` and now uses the exact matcher so adjacent ids survive cleanup.
- `pruneRolledJobLogSegments(ttl)` plus the new `startPrune(intervalMinutes, ttl)` API form a running-state remover that only deletes rolled segments whose last-modified age exceeds the configured retention window. Active files and unclassified sidecars are never touched here.
- A dedicated daemon `ScheduledExecutorService` (named `task-log-prune`) is wired into `SeaTunnelServer.initInternal`. It is driven by the new `TelemetryLogsConfig.jobLogPruneIntervalMinutes` setting (config key `job-log-prune-interval-minutes`, default 60 minutes) and uses a default TTL of 7 days to mirror the Log4j2 `file_ttl` policy. `SeaTunnelServer.shutdown(boolean)` calls `TaskLogManagerService.shutdown()` so the pruner stops with the rest of the engine services.
- The Log4j2 job appender is promoted from `File` to `RollingFile` with both `TimeBasedTriggeringPolicy` and `SizeBasedTriggeringPolicy` so long-running streaming jobs emit `.yyyy-MM-dd-N` segments that the pruner can shed independently. The `fileAppender` system route is untouched.
- New `ServerConfigOptions.JOB_LOG_PRUNE_INTERVAL_MINUTES` and `YamlSeaTunnelDomConfigProcessor` parsing so deployments can actually tune the prune cadence instead of inheriting the hard-coded default.
### Scope explicitly excluded from this PR
The original internal PR also touched whaletunnel-private classes and the log-download/extract protocol. Those pieces are intentionally **not** included here because they do not exist on `apache/seatunnel` `dev`:
- `LogoutService` and `LogServiceTest` (whaletunnel-private, used for log packaging and download).
- `JobClient.extractLogContentFromZip` and `JobClientLogStatusTest` (rolled-zip stitching for the download API).
- `log4j2_pod.properties` and `log4j2-test.properties` (whaletunnel-private deployment/test resources).
When whaletunnel is ready to upstream the full set, those changes can land as a follow-up PR after the web reader protocol is upgraded to handle multi-file / archived segments.
### Does this PR introduce _any_ user-facing change?
Yes — but only as additive, opt-in behavior with full backward compatibility.
- A new YAML key `job-log-prune-interval-minutes` (default `60`) under `seatunnel.engine.config.telemetry.logs` controls the new local job-log pruner. Setting it to a non-positive value disables pruning and preserves the current behavior. No existing keys are renamed or removed.
- The Log4j2 job appender now rolls segments by date and size. The active file name, the directory layout, and the routing key (`ST-JID`) are unchanged. Existing log consumers and the existing terminal `clean(jobId)` behavior continue to work as before; rolled segments are simply an additional output.
- The legacy 3-segment `MDCContext` serialization is still parsed and produced when callers do not opt into job mode, so existing log analysis pipelines and persisted context strings remain valid.
- A missing or unknown job mode fails closed to `unclassified` routing; the engine never guesses `STREAMING`. The PR does not modify the database schema, connector / datasource registration, REST API contracts, or `seatunnel-engine-ui`.
If a downstream deployer has a custom `log4j2*.properties` that copies `appender.routing.route.job.appender.type = File`, they will now see `RollingFile` instead once they sync to this commit. Migrating custom configs is a one-line change to `RollingFile`.
No entry is needed in `docs/en/introduction/concepts/incompatible-changes.md` because no public contract, default value, or configuration key is removed or renamed.
### How was this patch tested?
Validated on top of `apache/seatunnel` `dev` HEAD `00ca1bc02d`:
```bash
./mvnw -q -pl seatunnel-api,seatunnel-engine/seatunnel-engine-common,seatunnel-engine/seatunnel-engine-server -am spotless:apply
```
```bash
./mvnw -q -pl seatunnel-api,seatunnel-engine/seatunnel-engine-common,seatunnel-engine/seatunnel-engine-server -am -DskipTests -Dmaven.gitcommitid.skip=true install
```
```bash
./mvnw -q -pl seatunnel-api -Dtest=MDCTracerTest -Dmaven.gitcommitid.skip=true test
```
```bash
./mvnw -q -pl seatunnel-engine/seatunnel-engine-server -Dtest=JobLogFileNameMatcherTest,TaskLogManagerServiceTest,CoordinatorServiceJobLogModeTest -Dmaven.gitcommitid.skip=true test
```
```bash
./mvnw -q -pl seatunnel-api -Dmaven.gitcommitid.skip=true test
```
All commands above finish with exit code 0. The unit tests added by this PR exercise:
- Round-trip serialization of the legacy 3-segment format and the new 4-segment mode-aware format in `MDCContext`.
- Nested MDC activation: an outer mode-aware context activates `ST-JOB-MODE`, an inner context without mode explicitly clears it, and on deactivate the outer mode is restored.
- Exact filename matching for active, unclassified sidecar, and date-index rolled files; rejection of adjacent ids, `.tmp` suffixes, and bare date suffixes; `isRolledSegment` rejecting active and unclassified.
- `TaskLogManagerService.clean` deletes only files that exactly belong to the supplied job id and survives missing log directories and non-positive job ids.
- `CoordinatorService.extractJobMode` returns the configured mode for a complete payload and falls back to null when job config or job context is missing.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License Notice according [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md) — N/A, no new dependencies introduced.
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs — the only new user-facing surface is the optional `job-log-prune-interval-minutes` YAML key. Happy to add a short docs entry as a follow-up if reviewers prefer it land in this PR.
* [x] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR. — N/A, fully backward compatible.
* [x] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) — N/A, no connector.
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml) — N/A.
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml) — N/A.
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/) — N/A, this PR is engine-internal.
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config) — N/A.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Prepare job log mode context and exact filename matching [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11702:
URL: https://github.com/apache/seatunnel/pull/11702#issuecomment-6158102536
Quick follow-up on my approval of this head (`3f9d7a8648`), because the CI fact I cited earlier ("pending") has changed. There is no new commit and nothing in the diff changed, so my source-level conclusion stands: no code-side blocker. This is a static note; I did not execute the PR.
The fork `Build` run finished red, but only on `all-connectors-it-2` (JDK 11 and 17): `CassandraIT.testCassandra` and `testCassandraMultiTableSource` fail with `expected: <0> but was: <1>` on the Flink 1.20 variant. This PR does not touch Cassandra or the Flink 1.20 container, and the same two tests are what #12756 ("Prevent direct memory exhaustion in Flink 1.20 Cassandra tests") fixed on `dev` after this run. The branch is `diverged` (ahead 3, behind 10) and does not contain that fix yet.
Minimal next step: sync the latest `dev` into the branch, then re-run only the failed `all-connectors-it-2` jobs. If anything other than the Cassandra tests fails after that, please paste the new check link and I will look at it. A committer approval is still required because my account is comment-only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature][Zeta] Prepare job log mode context and exact filename matching [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11702:
URL: https://github.com/apache/seatunnel/pull/11702#issuecomment-6160492249
Thanks for the rebase. The current head (`3f9d7a8648`) is content-identical to the version I reviewed earlier (`533a1e4`): exact `JobLogFileNameMatcher` matching replacing the `contains` check, the `RollingFile` appender, the `startPrune` TTL pruner, and the `MDCContext` `JobMode` dimension with the legacy 3-segment format still round-tripping. From the code side I don't see a blocker.
The only open item is CI: the fork `Build` run is red on `all-connectors-it-2` (JDK 11 and 17), where `CassandraIT.testCassandra` and `testCassandraMultiTableSource` fail on the Flink 1.20 variant. This PR doesn't touch Cassandra or Flink 1.20, and those same failures were already fixed on `dev` after this run; the branch is behind `dev` and doesn't carry that fix yet.
Suggested next steps:
1. Sync the latest `dev` into your branch (no code changes expected).
2. Re-run the failed `all-connectors-it-2` jobs and drop the check link here.
3. If anything other than the two Cassandra tests fails after the sync, paste the failing test names/log and I'll take a look.
Note that a committer approval is still required for merge, as my account is comment-only.
<!-- streview-comment:1746 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org