You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/08/05 13:20:42 UTC
[I] [Feature][Zeta] Add a task exception and failure history view [seatunnel]
DanielLeens opened a new issue, #11667:
URL: https://github.com/apache/seatunnel/issues/11667
## Background
SeaTunnel already displays the current error on the job page, but only as a single raw string: `job.errorMsg` is rendered as one unstructured `<pre>` block (`seatunnel-engine-ui/src/views/jobs/detail.tsx`), with no per-task attribution and no history. Internally the story is the same: `PhysicalPlan.errorBySubPlan` is an `AtomicReference<String>` that uses `compareAndSet(null, ...)` to capture only the *first* error for a sub-plan and silently discards every error after that (`seatunnel-engine/.../dag/physical/PhysicalPlan.java`). `TaskExecutionState` does carry a `throwableMsg` per status report, but it is transient and never persisted per attempt, and there is currently no restart/attempt counter anywhere in the engine (`restartCount`/`attemptNumber`/`retryCount` do not exist).
Closely related: every pipeline and task group already maintains a timestamped state-transition history (CREATED to DEPLOYING to RUNNING to FINISHED/CANCELING/CANCELED/FAILING/FAILED) in a `stateTimestamps` array backed by a shared internal map (`SubPlan.java`, `PhysicalVertex.java`), but only the **job-level** timestamps are ever read back (`JobMaster#getStateTimestamp`); the pipeline- and task-group-level timestamps are recorded and then never exposed anywhere. That is exactly the data a "when did this task start failing, and how does that compare to its previous attempt" view would need, and it already exists in memory today.
Flink's Web UI addresses both needs with its "Exceptions" tab: the root exception plus the full exception history across restarts, each entry timestamped and attributed to a task/host.
## Problem to solve
Because only the first error per sub-plan is kept and there is no attempt concept, diagnosing an intermittently-failing job means either catching the error in real time before something else overwrites the single stored value, or digging through raw logs across every affected worker by hand. This makes it hard to tell a one-off failure apart from a recurring pattern.
## Proposed scope
Add a dedicated exception/failure history view, scoped to a job:
- persist more than the first error per sub-plan/task: keep a bounded history of failures (timestamp, failing task/subtask, host/worker, exception type and message)
- introduce an attempt/restart counter so history entries can be grouped by attempt, since none exists today
- expose the already-tracked-but-unread pipeline/task-group `stateTimestamps` alongside each failure entry, so a failure can be shown against how long that attempt had been running rather than as an isolated event
- apply to both running and finished jobs, within Zeta's existing history retention window
- link each entry to the corresponding task's log (building on #9050/#11662)
## Why this should be a dedicated feature
This requires retaining more than "the current error," which has retention, storage, and API-contract implications:
- how many historical failures are kept per job, and for how long after the job finishes
- how this interacts with existing finished-job history storage (including the known S3-backed finished-jobs issue, #10039) so it does not regress at scale
- exposing the pipeline/task-group `stateTimestamps` is extending an existing internal model to a new read path, not new data collection, but it still needs a defined contract
## STIP requirement before implementation
Because this changes what failure data Zeta retains and exposes as a stable contract, **the claimant should submit a STIP design first and get maintainer agreement before starting implementation**. The STIP should clarify:
- the exception-history data model, its retention limits for running vs. finished jobs, and how attempts are numbered
- how large exception messages/stack traces are truncated or paginated
- how this scales for jobs stored in external history backends (e.g. S3), consistent with the constraints already surfaced in #10039
- which fields are stable API/UI contract versus best-effort telemetry
## Acceptance criteria
- Users can see the full sequence of failures/restarts for a job, not only the latest one, from both the running and finished job views.
- Each entry identifies the failing task/attempt/host and links to its log.
- Recurring root causes across attempts are visually distinguishable from one-off failures.
- English and Chinese docs are updated.
## Non-goals for the first version
- automatic root-cause classification or suggested fixes
- indefinite retention of full stack traces for all historical jobs
- cross-job failure correlation/analytics
## Related work
- Issue #9105 (closed) - exception message formatting
- Issue #9050 (closed) - worker log file viewing
- Issue #11662 - job log links currently broken
- Issue #10039 (closed) - finished-jobs history at scale on S3
- Issue #11351
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add a task exception and failure history view [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11667:
URL: https://github.com/apache/seatunnel/issues/11667#issuecomment-5235424637
@davidzollo @DanielLeens I traced the current task failure and finished job history paths and prepared the design-first branch here:
#11734
The proposal uses pipeline-level attempt numbering, keeps the records in HA-backed job storage and applies the existing finished job history expiration after completion. History is bounded to 100 records per job and duplicate terminal state delivery is handled using pipeline attempt and task group identity.
I kept task id worker and exception type optional where the current transport cannot provide them reliably instead of deriving them from formatted stack traces.
This branch is docs only for now. Once the attempt storage retention and REST contracts are agreed I can start the backend implementation and keep the UI as a separate follow-up. Let me know. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add a task exception and failure history view [seatunnel]
Posted by "LeonxLJX (via GitHub)" <gi...@apache.org>.
LeonxLJX commented on issue #11667:
URL: https://github.com/apache/seatunnel/issues/11667#issuecomment-5493476525
I'd like to take this one (`[Feature][Zeta] Add a task exception and failure history vie`). I'll look into the root cause and follow up with a PR. (claiming via @LeonxLJX)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add a task exception and failure history view [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11667:
URL: https://github.com/apache/seatunnel/issues/11667#issuecomment-5235477560
I opened the formal STIP for this design:
STIP: #11735
Design PR: #11734
The STIP now also covers stack trace truncation and external finished job history backends. I will keep the PR design-only and wait for agreement on the contract before starting the engine implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add a task exception and failure history view [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11667:
URL: https://github.com/apache/seatunnel/issues/11667#issuecomment-5229109945
This reads like a useful Zeta observability feature rather than a bug.
The important part is that the proposal is not just "show the latest error better". It is really asking for a stable failure-history contract: multiple failure entries, attempt grouping, task attribution, timestamps, and retention behavior for finished jobs. That is why treating it as a dedicated design-first feature makes sense.
The strongest part of the issue is that most of the raw ingredients already exist in some form today:
- task and sub-plan failures are already observed,
- state-transition timestamps are already recorded,
- the gap is mainly that only the first error is retained and none of that history is exposed as a structured read path.
Before implementation starts, the key things to pin down are:
1. how many failure-history entries are retained and for how long;
2. whether attempt/restart grouping becomes part of a stable public contract;
3. how this interacts with finished-job history storage so we do not create a feature that only works reliably for running jobs.
The direction itself looks reasonable to keep open and continue refining. Since this is already labeled `help wanted`, contributions are very welcome once the design boundary is explicit enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add a task exception and failure history view [seatunnel]
Posted by "LeonxLJX (via GitHub)" <gi...@apache.org>.
LeonxLJX commented on issue #11667:
URL: https://github.com/apache/seatunnel/issues/11667#issuecomment-5492523817
I'd like to take this one (`[Feature][Zeta] Add a task exception and failure history vie`). I'll look into the root cause and follow up with a PR. (claiming via @LeonxLJX)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add a task exception and failure history view [seatunnel]
Posted by "LeonxLJX (via GitHub)" <gi...@apache.org>.
LeonxLJX commented on issue #11667:
URL: https://github.com/apache/seatunnel/issues/11667#issuecomment-5493501849
I'd like to take this one (`[Feature][Zeta] Add a task exception and failure history vie`). I'll dig into the root cause and follow up with a PR shortly. (claiming via @LeonxLJX)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org