You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/08/10 03:05:35 UTC

[PR] [Feature][Zeta] Add task failure history design [seatunnel]

goutamadwant opened a new pull request, #11734:
URL: https://github.com/apache/seatunnel/pull/11734

   <!--
   
   Thank you for contributing to SeaTunnel! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   ## Contribution Checklist
     - Make sure that the pull request corresponds to a [GITHUB issue](https://github.com/apache/seatunnel/issues).
     - Name the pull request in the form "[Feature] [component] Title of the pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
     - Minor fixes should be named following this pattern: `[hotfix] [docs] Fix typo in README.md doc`.
   -->
   
   ### Purpose of this pull request
   
   Refs #11667.
   
   This PR documents the proposed backend contract for retaining and exposing task failure history in Zeta Engine.
   
   The design defines:
   
   - pipeline-level attempt numbering across restores
   - structured task-group failure records
   - HA-backed storage for running and finished jobs
   - bounded retention and deterministic eviction
   - duplicate terminal-state handling
   - a REST contract for querying failure history
   - compatibility with the existing `errorMessage` field
   - acceptance criteria and implementation slices
   
   English and Chinese documentation are included.
   
   This PR is intentionally limited to the design. Engine storage, failure capture, REST API, and Web UI changes will follow after the contract is agreed.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. It adds English and Chinese documentation describing the proposed task failure history contract.
   
   It does not change current engine behavior or expose a new runtime API.
   
   ### How was this patch tested?
   
   Validated the sidebar configuration and checked the patch for whitespace errors.
   
   ```shell
   node --check docs/sidebars.js
   git diff --check
   ```
   
   No runtime tests were added because this PR only introduces design documentation.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License Notice according
     [New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [x] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the following files are updated:
     1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
     2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5846257666

   Thanks for taking another pass at this, @SEZ9 — and sorry, I think there's a genuine mismatch here worth untangling before it turns into more review cycles.
   
   I went back to the document at this exact head (`a986eb165c`) rather than relying on my own summary from yesterday, and all four of your points are there as explicit prose — they're just not inside the new "Normative Rules" (R1-R5) section, which is why I think they didn't show up when you focused your re-read there. Citing the actual lines at this head:
   
   1. **Auth/authz** — `## Security and Input Validation`, "Authentication" paragraph (`:395-397`): "The endpoint sits under the existing `JobInfoServlet` mappings, so it inherits the `BasicAuthFilter` boundary... It adds no endpoint-specific authentication," plus explicit statements that basic auth is disabled by default, the boundary is a single shared credential with no per-job/per-tenant authorization, and narrower access must be enforced at the gateway/network layer. That's the explicit statement you asked for in point 1.
   2. **Redaction before HA persistence** — same section, "Redaction" paragraph (`:409`): "Best-effort redaction runs... before every write to the running map or the finished snapshot," covering every capture path (worker-reported, deployment, node-loss, master-recovery, terminal-snapshot). That's your point 2.
   3. **Worker `host:port`** — "Worker addresses" paragraph (`:441`): "The first version neither stores worker `host:port` attribution fields in running or finished history nor exposes them in REST responses." Also stated in Scope (`:33`) and Problem (`:43`). That's your point 3.
   4. **`limit`/path validation** — `## REST Contract` (`:370-378`): the exact URI grammar (`^/([0-9]{1,19})/failures$`), the `limit` regex (`^[0-9]{1,10}$`), default/cap of 100, and the `400`/`404` rules for malformed input, plus decision table R5 for job resolution across the absent-state window. That's your point 4.
   
   So I don't think these are open gaps in the current head — the substance has been there since the round I cited it in my 2026-08-22 comment (under different line numbers, before the document's later restructuring), and it's been re-verified in my 2026-09-21 and 2026-09-25 reviews.
   
   What I think is a fair ask, though: fold the Security paragraphs into a table with the same shape as R1-R5 (writer/enforcement-point/fence style), and extend R5 (or add a sibling table) for the `limit`/path grammar specifically, so all the normative content lives in one consistent format instead of half in prose and half in tables. @goutamadwant, that seems like a reasonable, purely organizational follow-up rather than new content — happy to take a look once it lands.
   
   @SEZ9, if you still read any of these four as substantively unaddressed once you've re-read the actual prose at the lines above (not just the new table section), I'd genuinely like to know which one and why — that's a different, real conversation than "not yet written down."


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5350156887

   Thanks for the summary, @goutamadwant — confirming from my side: I re-reviewed the doc as of the current head (86dcca4) and all of the points I raised across the earlier CHANGES_REQUESTED rounds (attempt numbering, dedup via the job-scoped EntryProcessor, HA-store-acknowledged failover semantics, the 1 MiB truncation/retention contract, TaskDeployState exception propagation, and the existing errorMsg compatibility guarantee) are addressed, which is why my last review is APPROVED.
   
   One clarification on the remaining gate: reviewDecision on this PR is still REVIEW_REQUIRED and mergeStateStatus is BLOCKED because @SEZ9'\''s CHANGES_REQUESTED review from Aug 13 hasn'\''t been updated or dismissed yet — GitHub branch protection doesn'\''t clear that automatically just because CI is green and a later review approved. My own GitHub permissions here are comment-only, so I can'\''t dismiss another reviewer's request or change the merge gate myself. This genuinely needs @SEZ9 (or another maintainer with write access) to either re-review the current head or explicitly dismiss the stale request before this can merge. Nothing further needed on the design side from my end.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5302980101

   @DanielLeens Updated the English and Chinese design docs and STIP #11735
   
   The contract now includes the 1 MiB aggregate text budget, separate failure-history truncation rules, default Hazelcast map configuration, an independent terminal TTL and TaskExecutionState serialization compatibility
   
   Docs syntax and diff checks pass locally


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5323234679

   @DanielLeens updated the deployment failure contract in both the design PR and STIP-33
   
   TaskDeployState.failed now extracts the exception class message and bounded stack trace as strings on the worker. A live connector exception does not cross the Hazelcast response so the master does not need the connector exception class


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5379740886

   @goutamadwant understood, and sorry this is dragging — but the mechanics haven't changed since my Aug 20 comment: my GitHub permissions here are comment-only, so I have no way to re-review as @SEZ9, dismiss their CHANGES_REQUESTED, or override the branch-protection gate myself. Only @SEZ9 re-submitting a review on the current head, or a maintainer with write access using GitHub's "Dismiss review" action, can clear `reviewDecision: REVIEW_REQUIRED`. That's not something I can do mechanically from here.
   
   What I can do is give you something concrete to point @SEZ9 to when you ping them, since a bare "please re-review" request is easy to deprioritize. I diffed their Aug 13 CHANGES_REQUESTED (at `f8b61d52ffe2`) against the current head (`86dcca4f`) directly in the doc, issue by issue:
   
   - **Issue 1 (blocking) — no auth/redaction requirement on the REST endpoint**: now covered by the "Security and Input Validation" section — `docs/en/engines/zeta/task-failure-history.md:138-140` requires the same `BasicAuthFilter` boundary as the existing job-detail endpoints, and line 142 requires redaction before HA persistence (not just at the REST response).
   - **Issue 2 — unredacted payloads persisted long-term in HA state**: line 89 now requires sanitizing/bounding content at the capture boundary before it's written to HA or finished-job history, and Acceptance Criteria item 13 (line 184) makes this explicit.
   - **Issue 3 — raw worker `host:port` leaking topology**: line 150 now states worker addresses are optional and follow the same authorization boundary as the rest of the record.
   - **Issue 4 — `JobInfoServlet` routing/`limit` validation underspecified**: line 134 now requires exact-path-shape matching (no prefix/substring fallthrough) and line 144-146 requires malformed `jobId`/non-numeric `limit` to return a controlled `400` with no stack trace — matching Acceptance Criteria items 14 and 18 (lines 185, 189).
   
   So on the substance, all four of @SEZ9's findings do appear addressed in the current head, not just asserted — that's an independent read from me, not a decision on their behalf. But only @SEZ9 (or a maintainer) can act on it to actually clear the gate. I'd suggest re-pinging them with this line-by-line mapping attached so they can confirm quickly rather than re-reading the whole doc from scratch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5756165775

   > Rechecked the process feedback: #11735 is the canonical STIP discussion, and this design-only PR is not an approved implementation. The retention proposal follows history-job-expire-minutes and keeps the response additive. I will not treat green CI as design consensus. Would you prefer closing this design-only PR once the issue/dev-list record is sufficient, or retaining it as a draft reference?
   
   I think this can be a draft PR. Once the STIP PR has concrete code implementation, we can change the PR status to Ready and proceed with review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5842995250

   Thanks for the update on `a986eb165c`. I read through the new "Normative Rules" section (R1–R5) and the explicit create-before-capture ordering statement — that tightens the design considerably compared with `e944f1e3`, and the decision tables make the contract much easier to verify.
   
   That said, the four security points I raised on `docs/en/engines/zeta/task-failure-history.md` in my previous review still appear to be open in the current head, and I could not find them covered by R1–R5:
   
   1. **HIGH – unauthenticated exposure of raw exception messages / stack traces via `/job-info/{jobId}/failures`.** The contract still needs to state the authentication/authorization expectation for this endpoint and a redaction requirement for the persisted `errorMsg`/stack-trace payload before it is returned to API consumers. If the intent is to inherit whatever protection the existing `JobInfoServlet` routes have, please say so explicitly and call out that raw stack traces are a new class of data compared with the existing single `errorMsg` field.
   
   2. **MEDIUM – unredacted failure payloads persisted in the HA-backed Hazelcast IMap.** The per-map ownership/fence/expiry table in R4 is a good place to add a sanitization rule (what is stripped before write, and a size cap on the stored message/stack trace), so secrets that leak into an exception message are not replicated across cluster state for the full retention window.
   
   3. **MEDIUM – worker `host:port` in the REST response.** Please either make this field optional/configurable, or add a rationale in the doc for why cluster topology must be disclosed to every API consumer. Exposing it only to the same audience that can already list cluster members would be an acceptable resolution if documented.
   
   4. **LOW – input validation for the servlet routing extension and `limit`.** The doc should spell out the accepted range/default for `limit`, the behaviour on non-numeric or negative values, and how the `{jobId}/failures` path segment is parsed and rejected when malformed, so the implementation has a normative rule to follow rather than an implicit one.
   
   Concrete asks before I can approve:
   - Add a "Security" (or equivalent) normative section addressing points 1–3 with the same table style as R1–R5.
   - Add the `limit`/path validation rules from point 4, ideally as an extension of R5.
   
   Happy to take another look as soon as those land.
   
   <!-- streview-comment:1329 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5386318612

   Hi @goutamadwant 
   Good job
   The STIP proposal should be presented in an issue or issue comments (or on the dev@seatunnel.apache.org mailing list), rather than drafting design proposals separately via PRs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5308898520

   @DanielLeens Updated the design for both review points. Deployment failures now preserve the original cause through TaskDeployState and TaskGroupDeployException. Attempt advancement is serialized through the job-scoped EntryProcessor and completes before restored execution starts. EN and ZH docs and acceptance criteria are in sync.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5349978912

   The current head is green and Daniel’s latest review is approved. @SEZ9 could you refresh the older review state when you get a chance? The findings from that review are addressed on the current head.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5378186413

   @DanielLeens The current head is green and was approved with the updated design, but the @SEZ9 requested review still blocks the PR. Could the current head be re-reviewed, or the stale review be dismissed if its points are already covered?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5249098971

   @DanielLeens Updated the English and Chinese design docs based on the STIP feedback. The attempt, deduplication, truncation, finished-job retention, and existing errorMsg compatibility contracts are now explicit. The branch is also synced with current dev.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5299607328

   @DanielLeens updated the English and Chinese design docs and the STIP for the review points. Dedup is now scoped to retained records, failover only guarantees HA-store-acknowledged writes, and worker metadata is limited to the existing pending-job address value. I also removed the external history backend wording.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5389998279

   Thanks @davidzollo — that's a fair process point, and I'll defer to it. Since this has been evolving into a full STIP-style design document through several review rounds on this PR thread, it probably makes sense to move the remaining design discussion to an issue thread (or the dev@ mailing list) so it gets proper visibility, per the usual STIP process. I'll hold off on further detailed design-doc review rounds here until that's sorted out.
   
   One independent item still open from my last comment: the `@SEZ9` CHANGES_REQUESTED review is still blocking `mergeStateStatus`, and only a fresh review on the current head (or an explicit dismissal by a maintainer with write access) can clear that gate — that's unrelated to the STIP-process question above and will need to be resolved either way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5748015076

   Rechecked the process feedback: #11735 is the canonical STIP discussion, and this design-only PR is not an approved implementation. The retention proposal follows history-job-expire-minutes and keeps the response additive. I will not treat green CI as design consensus. Would you prefer closing this design-only PR once the issue/dev-list record is sufficient, or retaining it as a draft reference?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5287937715

   @DanielLeens @SEZ9 updated the English and Chinese design docs and STIP-33. The EntryProcessor update is now explicitly asynchronous from the Hazelcast operation thread, the new state map cannot use broader backup or persistence settings than existing job-state maps, and REST routing only accepts the two exact path shapes. Added acceptance criteria for all three cases. Let me know. thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5770149255

   @DanielLeens @davidzollo The EN/ZH design and canonical STIP #11735 are now aligned on cleanup, retention, failover recovery, late-write fencing and REST compatibility. Worker-address attribution is excluded from both storage and responses.
   
   Documentation consistency checks passed. Keeping this PR as a draft as requested; these updates define the implementation requirements, not a runtime-verified feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5755225778

   Thanks @goutamadwant for following up, and for taking @davidzollo's process point on board.
   
   On your question: since #11735 is the canonical STIP discussion, my preference is to close this design-only PR once the security points below have been carried over into that issue (or the dev@ list), rather than keeping two places for the same design.
   
   The items I'd like to see explicitly addressed in the #11735 record:
   
   1. `GET /job-info/{jobId}/failures` — the contract still needs a stated authentication/authorization expectation and a redaction requirement for exception messages and stack traces before they leave the server.
   2. Persistence of failure records in the HA-backed Hazelcast IMap — the capture-time redaction/truncation should be spelled out as a requirement for what is written to the IMap, not only for what the REST layer returns, since the retention window (following `history-job-expire-minutes`, as you note) means this data lives across the cluster for a while.
   3. Worker `host:port` in the response — please either drop it from the default response or make it explicitly opt-in / operator-only, and note the rationale in the design.
   4. `JobInfoServlet` routing and the `limit` parameter — please add concrete validation rules (allowed range, default, behaviour on invalid input) to the contract.
   
   If you add those to #11735 and drop a link here, I'm happy for this PR to be closed. Agreed that the substance needs to be settled on the issue side.
   
   <!-- streview-comment:1216 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5276137745

   @SEZ9 @DanielLeens I updated both design docs based on your latest comments.  
   redaction now happens before HA and finished history persistence and the REST section defines the existing auth boundary and controlled input validation.  
   the job scoped update now uses a Hazelcast EntryProcessor for atomic dedup sequence allocation append and eviction  
   could you please take another look? thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5261913150

   @DanielLeens I updated the design after the last review
   
   attemptStartedAt is now explicitly optional. The storage section follows the existing JobHistoryService and Hazelcast lifecycle without assuming a pluggable history backend. The routing requirement for /job-info/{jobId}/failures is documented in both English and Chinese.
   
   Could you please confirm if the design is ready for the first backend slice? thanks! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-5305613076

   @DanielLeens Updated the design around the actual Zeta failure paths. Diagnostic attempt identity is now separate from pipelineRestoreNum, worker failures are captured at PhysicalVertex, and deployment failures use the makeTaskGroupFailing path. I also defined attempt start time, unknown-job behavior, and both retention limits in EN and ZH.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Zeta] Add task failure history design [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on PR #11734:
URL: https://github.com/apache/seatunnel/pull/11734#issuecomment-6096343121

   Thanks for the detailed pointer back to the document at `a986eb165c`. If the `## Security and Input Validation` and `## REST Contract` sections contain the prose you describe — the `BasicAuthFilter` inheritance and shared-credential caveat, best-effort redaction before every write to the running map or finished snapshot, no storage or exposure of worker `host:port`, and the URI/`limit` grammar with `400`/`404` rules plus R5 — then that would cover the substance of PR11734-F1 through F4, and my earlier re-read focusing only on R1-R5 missed it. Apologies for the churn.
   
   Before I close these out, could you quote the relevant passages from the changed file at that head (or link the diff lines)? The thread so far only summarizes them, and I'd like to confirm against the document itself, particularly for F1 given its severity.
   
   One non-blocking suggestion to make this harder to miss in future rounds: fold the Security paragraphs into a table with the same writer / enforcement-point / fence shape as R1-R5, and either extend R5 or add a sibling table for the `limit`/path grammar, so all normative content lives in one place. Happy for that to be a follow-up if you prefer.
   
   <!-- streview-comment:1648 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org