[I] [Feature]: Lineage: expose the pipeline hop graph on lineage events (hop)

["bamaer (via GitHub)" <gi...@apache.org>]

bamaer opened a new issue, #9004:
URL: https://github.com/apache/hop/issues/9004

   ### What would you like to happen?
   
   The lineage hub (`org.apache.hop.lineage`) emits per-transform lifecycle, file-I/O and relational-I/O events, but no event carries the pipeline's hop graph, so a sink cannot tell which transform fed which. That matters because dataset lineage is derived *through* jobs: a job's accumulated inputs are paired all-to-all with its outputs.
   
   An OpenLineage sink has to choose between two broken options today:
   
   * one job per pipeline (what `plugins/tech/openlineage` does, so Marquez shows the datasets on the job a user opens): a pipeline that reads `a` and `b` and writes `x` and `y` produces four dataset edges (`a→x`, `a→y`, `b→x`, `b→y`) instead of two;
   * one job per transform with the I/O each transform emits: every edge disappears, because a reader's job has only inputs and a writer's job has only outputs (`dataset → dataset` lineage has to pass through a job that holds both ends).
   
   Declared `fineGrainedLineages` do not fix dataset-level pairing (measured: DataHub stores the declaration faithfully and still reports the full product at dataset level).
   
   **Request:** emit the pipeline's hop graph once per run, as an additive event kind:
   
   ```java
   public enum LineageEventKind { ..., PIPELINE_GRAPH }
   
   public final class PipelineGraphLineagePayload implements LineagePayload {
     public record Transition(String from, String to) {}
     private final List<Transition> transitions;   // enabled hops, in pipeline order
   }
   ```
   
   emitted from `LineageRunLifecycleEmitter.emitPipeline` when `phase == STARTED`, with the same context, built from `PipelineMeta.getHops()` filtered to `hop.isEnabled()` with both ends present. Roughly 80 lines including a test.
   
   With the graph a sink can:
   
   1. attach file/relational I/O to the transform's job instead of the pipeline's, and
   2. propagate: `inputs(T) = ownReads(T) ∪ ⋃ outputs(U)` for `U ∈ prev(T)`, transitively,
   
   which gives the true dataset edges for chains (including pass-through transforms that do no I/O of their own), fan-out and fan-in. The pipeline job stays the run-level entry point without datasets; the transform jobs carry the pairs.
   
   This is additive: no existing sink behaves differently until it reads the new kind. The mapping change in the OpenLineage sink would be gated by a flag and measured before it becomes a default, because it also moves the datasets from the pipeline job to its transforms in Marquez.
   
   ### Issue Priority
   
   Medium (needed before lineage can be correct for multi-input/multi-output pipelines; nothing else depends on it)
   
   ### Issue Component
   
   Engine, Plugins
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature]: Lineage: expose the pipeline hop graph on lineage events (hop)

["mattcasters (via GitHub)" <gi...@apache.org>]

mattcasters closed issue #9004: [Feature]: Lineage: expose the pipeline hop graph on lineage events
URL: https://github.com/apache/hop/issues/9004


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature]: Lineage: expose the pipeline hop graph on lineage events (hop)

["bamaer (via GitHub)" <gi...@apache.org>]

bamaer commented on issue #9004:
URL: https://github.com/apache/hop/issues/9004#issuecomment-6249671602

   .take-issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org