You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/08/05 13:20:36 UTC

[I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

DanielLeens opened a new issue, #11665:
URL: https://github.com/apache/seatunnel/issues/11665

   ## Background
   Zeta's REST API already returns rich per-node monitoring data via `GET /system-monitoring-information`: roughly 35 fields including CPU load (process/system/systemAverage), heap/physical/managed memory, GC counts and time, and thread counts (the full payload is already modeled in `seatunnel-engine-ui/src/service/manager/types.ts`). However, the Workers/Master page (`seatunnel-engine-ui/src/views/managers/index.tsx`) only renders 4 of those fields today - Host, Port, Physical MEM Total, and Heap MEM Used - and an Action column is present in the source but commented out.
   
   Separately, Zeta already computes a full task-to-worker assignment (`GET /trace/task-mapping/:jobId`, backed by `TaskMappingBuilder`) with per-task host/worker binding, but that endpoint has no UI consumer at all today.
   
   What is genuinely missing, not just unrendered, is per-worker **slot accounting**: `OverviewInfo` only exposes cluster-wide `totalSlot`/`unassignedSlot` integers; there is no per-node slot total/used breakdown anywhere in the current model.
   
   In Flink's Web UI, the "Task Managers" tab is the equivalent first-class page: per-TM memory, CPU, GC, and slot occupancy, plus the tasks currently scheduled on it.
   
   ## Problem to solve
   Job-level and pipeline-level observability (#11351/#11352/#11353) can tell you *that* a vertex is slow, but not *why* at the infrastructure layer - e.g. whether the host running that task is CPU-saturated, low on heap, or has no free slots left for retry/redeploy. Today that requires shelling into a node directly, even though most of the raw data is already being collected by the cluster itself.
   
   ## Proposed scope
   Extend the existing Workers/Master page into a proper node resource view, mostly by surfacing data that already exists:
   - render the full already-fetched `/system-monitoring-information` payload (CPU load, GC, thread count, etc.) instead of the current 4 columns
   - add per-worker slot accounting (total/used) to the node model and surface it in the same view - this is the one part that needs new server-side bookkeeping, since slots are currently only tracked cluster-wide
   - wire the existing `/trace/task-mapping` endpoint into this page so a worker row can expand into "tasks currently running here", linking back to the owning job
   - keep this primarily a UI + light REST-extension change, since the bulk of the raw resource data is already collected
   
   ## Why this should be a dedicated feature
   This is cluster/infrastructure-level observability, distinct from job/pipeline-level observability:
   - it is scoped by node, not by job or vertex, so it needs its own UI treatment and a small REST extension (per-worker slots)
   - it must keep working (and stay useful) even when no job is currently running
   - per-worker slot accounting must stay consistent with the live scheduler view, which is a correctness-sensitive contract, not just a display change
   
   ## STIP requirement before implementation
   Because this adds new per-worker slot-accounting state, **the claimant should submit a STIP design first and get maintainer agreement before starting implementation**. The STIP can stay narrow given how much already exists:
   - how per-worker slot total/used is derived and kept consistent with the scheduler's live view
   - whether `/trace/task-mapping` needs any changes to serve a per-worker "tasks running here" list efficiently, or is already sufficient as-is
   - refresh interval and cost on larger clusters for the now-fuller `/system-monitoring-information` payload
   
   ## Acceptance criteria
   - The Workers/Master page renders the full monitoring payload already returned by `/system-monitoring-information`, not just today's 4 fields.
   - Users can see per-worker slot usage and the tasks currently assigned to each worker, without SSHing into a host.
   - The view remains useful when the cluster is idle (no jobs running).
   - REST/API and UI behavior are documented clearly.
   - English and Chinese docs are updated.
   
   ## Non-goals for the first version
   - historical/long-term retention of node resource metrics
   - cluster capacity planning or autoscaling recommendations
   - per-node log tailing (already covered by the existing worker log viewer)
   
   ## Related work
   - Issue #11351
   - Issue #11352
   - Issue #11353
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "danielnadean (via GitHub)" <gi...@apache.org>.
danielnadean commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5235177831

   I would like to work on this issue.
   
   I will start with the STIP/design required above, focused on:
   - how per-worker slot total/used will be derived and kept consistent with the live scheduler view (the one genuinely new piece of state here)
   - whether `/trace/task-mapping` is sufficient as-is for the per-worker "tasks running here" drill-down, or needs a narrower read path
   - refresh interval/cost once the page renders the full `/system-monitoring-information` payload instead of today's 4 fields
   
   I'll open the STIP/design PR first and continue with implementation after maintainer alignment, following the same sequencing used on #11351.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5352360146

   working on this and raise a first slice PR shortly! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "danielnadean (via GitHub)" <gi...@apache.org>.
danielnadean commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5236826008

   @goutamadwant you're right, and I should have found this before opening PRs.
   
   Two things I missed in my own research:
   
   1. `WorkerResourceDiagnostic` (`org.apache.seatunnel.engine.server.diagnostic`) already models almost exactly this per-worker shape - `address`, `tags`, `totalSlots`, `freeSlots`, `dynamicSlot`, `cpuUsage`, `memUsage`, `runningJobIds`. I built a parallel `WorkerOverviewInfo` with different field names for the same concepts instead of reusing or aligning with it.
   2. #11597 is already adding a running-job-centric slot usage API. Adding a third, worker-centric slot representation on top without reconciling with either of those is exactly the kind of competing-model fragmentation you flagged.
   
   You're also correct on the substance, independent of prior art: my `totalSlot = assignedSlots.length + unassignedSlots.length` is wrong for dynamic-slot workers, for the reason you gave - `unassignedSlots` is not a fixed remaining pool for those, so summing the two array lengths does not mean "capacity."
   
   Withdrawing my claim and closing #11732 / #11733. Your proposed backend contract (fixed vs. dynamic slot representation, explicit snapshot/unavailable-worker semantics, reusing `WorkerResourceDiagnostic` where possible, alignment with #11597) is the right foundation - please go ahead with it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5747163473

   @goutamadwant thanks for following through on the agreed direction — the backend-first slice in #11909 matches what was converged on here: reusing the existing `WorkerResourceDiagnostic` shape instead of a third slot model, keeping fixed and dynamic slot semantics separate, sourcing values from the latest heartbeat, and returning an unavailable snapshot during master-election gaps rather than an error. The coverage you list (slot release/reuse, incomplete snapshots, master changes, serialization, REST output) is the right set for this contract.
   
   A few asks to close out the backend point before the UI work in #12398 builds on it:
   
   1. In #11909, please call out explicitly (in the PR description or a short doc note) how a dynamic-slot worker is represented in the response — i.e. what `totalSlots` / `freeSlots` mean for it and how a consumer tells it apart from a fixed-slot worker — so the UI does not fall back into the `assigned + unassigned` capacity assumption that @danielnadean withdrew.
   2. Confirm how the per-worker read path relates to the running-job-centric slot usage from #11597 — reuse, shared fields, or intentionally independent — so we don't end up with two diverging representations.
   3. Since #12398 depends on the #11909 contract, let's treat the backend PR as the one to settle first; if anything in the endpoint shape changes during its review, please mirror it in #12398 rather than letting the UI lock in an earlier form.
   
   Task-level drill-down staying separate is fine. Once the points above are answered on #11909 I'll take a look at both PRs.
   
   <!-- streview-comment:1181 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5239947694

   Thanks for converging this thread.
   
   I re-checked the current `dev` code before replying here. The feature gap is still real, and the narrower backend-first direction in the latest comments is the right one:
   
   - there is already an existing per-worker diagnostic shape in `WorkerResourceDiagnostic` (`address`, `tags`, `totalSlots`, `freeSlots`, `dynamicSlot`, `cpuUsage`, `memUsage`, `runningJobIds`);
   - the current overview read path still exposes only cluster-wide slot totals (`totalSlot` / `unassignedSlot`);
   - I do not see a merged per-worker REST read path on current `dev` yet.
   
   So this should stay open as a genuine node-resource / worker-slot observability feature, not as something already finished upstream.
   
   Given the new comments, the safest next step is:
   - reuse or align with the existing worker diagnostic / `WorkerProfile` concepts instead of introducing a third slot model;
   - represent dynamic-slot workers explicitly rather than treating `assigned + unassigned` as fixed capacity;
   - keep the UI as a follow-up once the backend read contract is stable.
   
   Since #11732 and #11733 were withdrawn, one backend-focused PR from the agreed direction now looks like the cleanest path forward, with tests around fixed-slot vs dynamic-slot workers and snapshot / unavailable-worker semantics.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5744378337

   Opened #12398 as the Web UI follow-up to #11909, adding worker resource details, fixed/dynamic slot display, and refresh support. Task-level drill-down remains separate.
   @DanielLeens 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5366247068

   Thanks @goutamadwant, this matches the direction we converged on in this thread — reusing the existing worker diagnostic model instead of introducing a third slot representation, keeping fixed vs. dynamic slot semantics separate, and returning an explicit unavailable snapshot during master-election gaps rather than an error. The test coverage you describe (slot release/reuse, incomplete snapshots, master changes, serialization, REST output) also lines up with what was asked for after #11732 / #11733 were withdrawn.
   
   I'll review #11909 in detail there. A few concrete things I'd like confirmed on the PR so review goes quickly:
   
   1. How the response represents dynamic-slot workers — specifically that we are not deriving a fixed capacity from `assigned + unassigned`, per the point @danielnadean conceded earlier.
   2. Where the endpoint aligns (or intentionally diverges) from the running-job-centric slot API in #11597, so the two read paths don't fragment again.
   3. That the exposed fields map cleanly onto the existing `WorkerResourceDiagnostic` shape (`address`, `tags`, `totalSlots`, `freeSlots`, `dynamicSlot`, `cpuUsage`, `memUsage`, `runningJobIds`) rather than renaming the same concepts.
   
   Agreed that the Web UI and task drill-down stay as follow-ups once this backend contract is settled. Nice work getting the first slice up quickly.
   
   <!-- streview-comment:428 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5357364614

   @DanielLeens @davidzollo The backend-first slice is ready for review: #11909 
   
   It reuses the existing worker diagnostic model and adds a read-only worker resource endpoint for both REST API versions. Fixed and dynamic slot semantics are kept separate, worker resource values come from the latest heartbeat, and master-election gaps return an unavailable snapshot instead of an error.
   
   I also added coverage for slot release and reuse, incomplete snapshots, master changes, serialization, and REST output. The Web UI and task drill-down remain follow-up work after this contract is agreed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5229110041

   This looks like a reasonable infrastructure-observability feature request rather than a runtime bug.
   
   The proposal is also scoped in a good way: most of the data is already available, and the one genuinely new contract appears to be per-worker slot accounting. That distinction matters, because surfacing existing monitoring fields is mostly a UI/read-path problem, while per-worker slot totals and usage become scheduler-facing correctness data.
   
   From a review perspective, the most important design questions are:
   1. how per-worker slot total/used will be derived and kept consistent with the live scheduler view;
   2. whether `/trace/task-mapping` is already sufficient for the "what is running on this worker" drill-down, or needs a narrower server-side read path;
   3. what refresh cost is acceptable on larger clusters once the page renders more of the monitoring payload continuously.
   
   So I would keep this open as a dedicated feature, with the implementation split kept narrow: first make the contract around per-worker slots explicit, then wire the existing monitoring/task-mapping data into the UI around that contract.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "danielnadean (via GitHub)" <gi...@apache.org>.
danielnadean commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5235305537

   Opened the STIP/design PR: https://github.com/apache/seatunnel/pull/11732
   
   Opened the implementation PR (draft, to be reviewed together with the design): https://github.com/apache/seatunnel/pull/11733


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "danielnadean (via GitHub)" <gi...@apache.org>.
danielnadean commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5236790893

   ## Design proposal (for discussion on this issue)
   
   Following the same pattern adopted on #11666 (see discussion there with @nzw921rx and @DanielLeens): keeping pre-implementation design alignment here on the issue instead of a standalone docs PR. User-facing docs will be updated together with the implementation PR once the feature ships, not merged ahead of it. Closing the earlier docs-only PR (#11732) accordingly.
   
   ### Existing foundation (why this can stay small)
   
   | Area | Existing contract |
   |---|---|
   | Per-node JVM/host metrics | `GET /system-monitoring-information` already returns ~35 fields per node (CPU load, heap/physical memory, GC, thread count, ...). The Workers/Master page renders only 4 of them today. |
   | Per-worker slot state | `ResourceManager#getRegisterWorker()` returns a live `ConcurrentMap<Address, WorkerProfile>`. Each `WorkerProfile` already carries `assignedSlots`/`unassignedSlots` (`SlotProfile[]`) and `systemLoadInfo` (`cpuPercentage`/`memPercentage`) - the same state the scheduler itself uses to accept/reject allocation requests. |
   | Cross-node read pattern | `GetOverviewOperation`/`OverviewService` already show the established local-vs-forward-to-master routing for reaching this state from REST. |
   
   So the "per-worker slot accounting" this issue originally asked for isn't new bookkeeping - it already exists inside `WorkerProfile`. The only new backend work is a read-only projection of it.
   
   ### V1 scope
   
   - Extend the Workers/Master table with a curated column set (Host, Port, Role, CPU Load, Heap Used/Max, Physical MEM, GC, Threads, Slots used/total) instead of today's 4 columns.
   - Re-enable the existing (currently commented-out) Action column as a "View Details" drawer showing the full raw monitoring payload, so no field is lost by curating the table.
   - New read-only `GET /resource/workers` endpoint projecting `WorkerProfile` -> `{host, port, totalSlot, usedSlot, dynamicSlot, cpuPercentage, memPercentage, attributes}`. No new field on `WorkerProfile`, no change to any allocation/scheduling code path.
   - Client-side join between `/system-monitoring-information` and `/resource/workers` by `host:port`.
   
   ### Non-goals for V1
   
   - Per-worker "tasks currently running here" via `/trace/task-mapping` - that endpoint is scoped per-job; a cluster-wide, cross-job view needs its own fan-out design and is left for a follow-up.
   - Historical retention of node resource metrics, capacity planning/autoscaling.
   
   ### Validation
   
   - Unit test coverage for the `WorkerProfile` -> projection mapping, including a worker with no slots/load reported yet (must not throw).
   - Frontend test coverage for the client-side join, including an unmatched row.
   
   Implementation PR: #11733 (draft; docs bundled in the same PR per the above).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Zeta] Add a worker node resource view to the Zeta Web UI [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11665:
URL: https://github.com/apache/seatunnel/issues/11665#issuecomment-5234523640

   Hi, I would like to take the first backend slice for this.\n\nI checked the current  code before proposing the scope.  already provides the node metrics,  and  already carry worker resource snapshots, and  maps running tasks to workers. PR #11597 is separately defining running-job slot usage, so I do not want to introduce another competing slot ownership model here.\n\nMy proposed first slice is:\n\n1. Define a read-only per-worker resource contract keyed by worker address.\n2. For fixed-slot workers, expose total, used, and free slots from one  snapshot.\n3. For dynamic-slot workers, do not present  as total capacity. Dynamic slots are created from the remaining resource, so the response should expose , used slot count, and total/remaining resource instead.\n4. Make the snapshot and unavailable-worker semantics explicit instead of combining coordinator and member data as if they were one atomic real-time view.\n5. Add focused tests for fixed and dynamic slots, released/reused slo
 ts, and missing or stale worker snapshots.\n6. Keep the UI as a follow-up. Once the backend contract is agreed, the UI can reuse the existing monitoring fields and task mapping, and align any job ownership fields with #11597 if that PR lands.\n\nDoes this backend boundary and the dynamic-slot representation look right before I start the implementation? If yes, I am happy to take it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org