Passive lineage collection?

[Maxwell Pospischil <ma...@cloudkitchens.com>]

Hey all,

We've got a number of flink jobs deployed and I'm wrestling with how best
to collect data lineage information from them. Our most common connector on
either end is Kafka, as either a source or a sink. I've found that I can
subclass the Kafka record de/serializers and collect topics off of the
messages on the way in/out for the jobs that use the DataStream api but I'm
finding it significantly more cumbersome to do similar for the ones going
through the sql connector.

I've taken a step-back from that approach and am wondering if maybe there's
a passive way I can collect this information by observing the flink job
externally? I've found my source topics sprinkled throughout the logs,
checkpoint files and in metric names on the job graph and if source topics
were all that I was worried about, I think I could collect them from any of
those places. I'm having a hard time finding sink topics by any passive
means though, is it possible to get that information anywhere?

Thanks!