You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "addu390 (via GitHub)" <gi...@apache.org> on 2026/06/11 23:58:59 UTC
[I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
addu390 opened a new issue, #841:
URL: https://github.com/apache/flink-agents/issues/841
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
Follow-up from #710.
The event log records every event with its full payload, but you can't rebuild a single run as a causal tree. A record has no link to the action that emitted the event or the event that action was processing.
Proposal: add a few lineage fields to each event log record (emitting action, source event id, run/trace id) so a run's trace tree can be rebuilt offline from the log. Reverse edges (event to triggered actions) can be derived from the agent plan's trigger rules, so no need to record them.
Scope is recording only, per #710. Reconstruction, visualization and OTel export are potential future follow-ups. I'll share a short design here before implementing.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "zxs1633079383 (via GitHub)" <gi...@apache.org>.
zxs1633079383 commented on issue #841:
URL: https://github.com/apache/flink-agents/issues/841#issuecomment-4889908286
I took a pass through the current event-log/runtime shape and I think this can be a narrow first PR if the lineage data stays on the log record/context rather than inside user event attributes.
Current insertion-point observation:
- `EventLogRecord` is already the right wire object for the log; it currently carries `EventContext` + `Event`.
- `EventContext` currently has only `eventType` and `timestamp`.
- `EventRouter.notifyEventProcessed(Event)` only receives the event, so it cannot know which action emitted it.
- `RunnerContextImpl.sendEvent(Event)` does know the current `actionName`, but that metadata is lost before the event reaches the logger.
My suggested first-PR shape:
1. Add a small lineage/context object to `EventLogRecord` or extend `EventContext` with optional fields:
- `logSchemaVersion`
- `runId` / `traceId`
- `eventId`
- `sourceEventId` or `parentEventId`
- `emittingAction`
- optionally the current key/sequence number if we can expose them without leaking unstable internals
2. Keep these fields outside `event.attributes`, so the event payload remains user-owned and cross-language JSON-compatible.
3. Treat each input event as the root of a run: `runId = inputEvent.id`, no parent/source event, no emitting action.
4. For events emitted from an action, set `runId` from the source event, `sourceEventId = actionTask.event.id`, and `emittingAction = action.name`.
5. Leave reverse edges out of the log, as proposed. They can still be derived from the agent plan's trigger rules during offline reconstruction.
This keeps the first implementation scoped to recording and replay/debuggability. Visualization, OTel export, and trace-tree reconstruction can be follow-ups once the log has stable causal fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "rosemarYuan (via GitHub)" <gi...@apache.org>.
rosemarYuan commented on issue #841:
URL: https://github.com/apache/flink-agents/issues/841#issuecomment-4901457669
Thanks for sharing the design direction @addu390 @zxs1633079383 . I took a look at the current runtime/event-log path, and one identity detail around `runId` seems worth settling before the schema lands.
Using `runId = inputEvent.id` may not be stable enough. In Java, `InputEvent` is freshly created in `wrapToInputEvent` with a random UUID, so source replay after failover can give the same logical input a different run id. In Python, event ids are content-derived, so two distinct inputs with identical payloads can share the same run id and merge into one run.
A more replay-stable option could be deriving `runId` from the stable job identifier, key, and per-key sequence number, for example `jobIdentifier:key:sequenceNumber` with a well-defined key encoding. This does not add a new identity concept: `key + sequenceNumber` is already part of the identity `ActionStateUtil.generateKey` relies on to match action state across failover.
I am currently working on Memory observability, which seems closely related to event-log trace work. Do you have any near-term plans to work on this issue? If not, I would be happy to pick it up after finishing Memory observability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 commented on issue #841:
URL: https://github.com/apache/flink-agents/issues/841#issuecomment-4911258233
Thanks to @addu390, @zxs1633079383, and @rosemarYuan for the great discussion.
I noticed a few detailed questions around lineage tracking and field design, so I think it would be helpful to align on the design before moving forward with development. @rosemarYuan, could you please post a detailed design proposal in the GitHub discussion for everyone to review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "rosemarYuan (via GitHub)" <gi...@apache.org>.
rosemarYuan commented on issue #841:
URL: https://github.com/apache/flink-agents/issues/841#issuecomment-4911788862
Thanks for assigning this to me. I’d be happy to take it on.
I agree it would be better to align on the field semantics and lineage model before starting the implementation. I’ll prepare a detailed design proposal in the GitHub discussion soon, covering the proposed field set, run identity derivation, causal edge semantics, and the scope of the first recording-only PR.
Looking forward to everyone’s feedback there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 closed issue #841: [Feature] Add lineage fields to the event log for per-run trace reconstruction
URL: https://github.com/apache/flink-agents/issues/841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "rosemarYuan (via GitHub)" <gi...@apache.org>.
rosemarYuan commented on issue #841:
URL: https://github.com/apache/flink-agents/issues/841#issuecomment-4904591952
Thanks @addu390, happy to take this up after I wrap up the current Memory observability work. I’ll keep the first PR scoped to recording-only and share the design/implementation for review.
Agreed on the split for @zxs1633079383:
- `eventId`: the concrete event id, preserving current event identity semantics.
- `runId`: a replay-stable run identity derived from stable runtime coordinates.
- `sourceEventId` / `emittingAction`: the causal edge from the processed event to the event emitted by an action.
One small addition I think would help reconstruction:
- `emitIndex`: the ordinal of an emitted event within the parent action invocation.
The reason is that `sourceEventId + emittingAction` identifies which action invocation emitted an event, but it does not order multiple sibling events emitted by that same invocation. Recording `emitIndex` makes the sibling order explicit, instead of relying on log layout.
This is useful because:
- Python generator/async actions can drain emitted events across multiple polls.
- Per-subtask event-log files may later be merged, where cross-file line order is not a strong contract.
- `(sourceEventId, emittingAction, emitIndex)` gives v1 a stable emitted-event occurrence key and leaves a clean path for a later deterministic event-address scheme.
I would define `emitIndex` as the ordinal within the parent action invocation, rather than just the local batch index.@zxs1633079383 thanks for the field shape, would welcome your review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "zxs1633079383 (via GitHub)" <gi...@apache.org>.
zxs1633079383 commented on issue #841:
URL: https://github.com/apache/flink-agents/issues/841#issuecomment-4902334354
Thanks @rosemarYuan, good catch. I agree that `runId = inputEvent.id` is not the right stable identity.
I also checked the current Java/Python event identity behavior:
- Java input events can get a fresh random UUID when wrapped, so replay/failover can change the id for the same logical input.
- Python event ids are content-derived, so identical payloads can collide across distinct logical inputs.
A cleaner split may be:
- `eventId`: the concrete event id, preserving current event identity semantics.
- `runId` / `traceId`: a replay-stable run identity derived from stable runtime coordinates, e.g. job identifier + key + per-key sequence number, with a defined key encoding.
- `sourceEventId` / `parentEventId`: causal edge from the event being processed to the event emitted by an action.
- `emittingAction`: the action that emitted the event.
For the first PR, I think the important regression tests are:
1. Java replay/wrapping should not produce a different run id for the same logical input position.
2. Python identical payloads under different key/sequence positions should not collapse into one run.
3. The lineage fields should stay outside user event attributes.
Since @addu390 opened the issue and offered to submit a PR, it would be good to confirm whether they still plan to take it. I am happy to help review the design/tests or contribute a focused follow-up if useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Add lineage fields to the event log for per-run trace reconstruction [flink-agents]
Posted by "addu390 (via GitHub)" <gi...@apache.org>.
addu390 commented on issue #841:
URL: https://github.com/apache/flink-agents/issues/841#issuecomment-4904343853
Hi @zxs1633079383 @rosemarYuan, thanks for the interest on this issue, I haven't started working on it quite yet. Feel free to take up the issue and implementation. I can help with the review of the design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org