You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@seatunnel.apache.org by Goutam <wo...@gmail.com> on 2026/08/10 03:18:01 UTC
[DISCUSS] STIP-33: Zeta task exception and failure history contract
Hi Seatunnel Community
I opened STIP-33 to propose a structured and bounded task failure history
for Zeta Engine.
STIP:
https://github.com/apache/seatunnel/issues/11735
Design PR:
https://github.com/apache/seatunnel/pull/11734
Related feature:
https://github.com/apache/seatunnel/issues/11667
Today the engine retains one formatted error message, which makes it
difficult to understand failures across pipeline restores or failures
reported by different task groups.
The proposal defines:
- pipeline-level attempt numbering
- structured task-group failure records
- HA-backed storage for running and finished jobs
- bounded retention and stack trace handling
- duplicate failure handling
- external finished-job history compatibility
- an additive REST contract
- a separate follow-up for the Web UI
I would particularly like feedback on:
1. whether pipeline attempt numbering should become part of the public
contract
2. whether 100 retained failures per job is a reasonable initial bound
3. how failure history should integrate with external finished-job history
backends
4. whether stack traces should be truncated in the initial response or
exposed through a separate detail endpoint
The current PR is design and documentation only. I will wait for agreement
on the STIP before starting the engine implementation.
Thanks
Goutam Adwant
Re: [DISCUSS] STIP-33: Zeta task exception and failure history contract
Posted by David Zollo <da...@gmail.com>.
Good STIP.
The direction is meaningful.
I have some suggestions as follows:
1. taskGroupId can't be required. Checkpoint-coordinator failures and
slot-allocation failures are pipeline-scoped — no task group ever reports
FAILED for them (SubPlan.handleCheckpointError, makePipelineFailing). These
are the classic restart-loop causes, and under the current contract they'd
produce climbing attempt numbers with an empty failure history. Needs an
optional taskGroupId plus a required scope discriminator (TASK_GROUP |
COORDINATOR | SCHEDULER).
2. Attempt number has a race at delivery time. pipelineRestoreNum
increments before redeploy, but a stale terminal-state report from the
prior attempt (especially via the JobRestoreInProgressException retry path)
can arrive after the counter has already moved. Reading "current attempt"
at delivery time both mis-attributes the old failure and lets the dedup key
silently swallow the new attempt's real failure. Attempt must be stamped at
deploy time and echoed back in TaskExecutionState, not read live.
3. Same-jobId resubmission may be breaks dedup and numbering. Savepoint
restart / custom-jobId resubmission restarts attempt at 0 while the prior
run's history (same jobId) can still be alive for up to
history-job-expire-minutes. First failure of the new run collides with
(pipelineId, 0, taskGroupId) from the old run and gets dropped.
By the way, I think 100 retained failures per job is a reasonable initial
bound.
Best Regards,
David
Linkedin: https://www.linkedin.com/in/davidzollo
On Sun, Aug 9, 2026 at 11:19 PM Goutam <wo...@gmail.com> wrote:
> Hi Seatunnel Community
>
> I opened STIP-33 to propose a structured and bounded task failure history
> for Zeta Engine.
>
> STIP:
> https://github.com/apache/seatunnel/issues/11735
>
> Design PR:
> https://github.com/apache/seatunnel/pull/11734
>
> Related feature:
> https://github.com/apache/seatunnel/issues/11667
>
> Today the engine retains one formatted error message, which makes it
> difficult to understand failures across pipeline restores or failures
> reported by different task groups.
>
> The proposal defines:
>
> - pipeline-level attempt numbering
> - structured task-group failure records
> - HA-backed storage for running and finished jobs
> - bounded retention and stack trace handling
> - duplicate failure handling
> - external finished-job history compatibility
> - an additive REST contract
> - a separate follow-up for the Web UI
>
> I would particularly like feedback on:
>
> 1. whether pipeline attempt numbering should become part of the public
> contract
> 2. whether 100 retained failures per job is a reasonable initial bound
> 3. how failure history should integrate with external finished-job history
> backends
> 4. whether stack traces should be truncated in the initial response or
> exposed through a separate detail endpoint
>
> The current PR is design and documentation only. I will wait for agreement
> on the STIP before starting the engine implementation.
>
> Thanks
> Goutam Adwant
>